Self-Driving Car Engineer Nanodegree

Deep Learning

Project: Build a Traffic Sign Recognition Classifier

In this notebook, a template is provided for you to implement your functionality in stages which is required to successfully complete this project. If additional code is required that cannot be included in the notebook, be sure that the Python code is successfully imported and included in your submission, if necessary. Sections that begin with 'Implementation' in the header indicate where you should begin your implementation for your project. Note that some sections of implementation are optional, and will be marked with 'Optional' in the header.

In addition to implementing code, there will be questions that you must answer which relate to the project and your implementation. Each section where you will answer a question is preceded by a 'Question' header. Carefully read each question and provide thorough answers in the following text boxes that begin with 'Answer:'. Your project submission will be evaluated based on your answers to each of the questions and the implementation you provide.

Note: Code and Markdown cells can be executed using the Shift + Enter keyboard shortcut. In addition, Markdown cells can be edited by typically double-clicking the cell to enter edit mode.


Step 0: Load The Data

In [1]:
# Load pickled data
import pickle

# TODO: Fill this in based on where you saved the training and testing data

training_file = 'train.p'        # saved into the project folder
testing_file = 'test.p'

with open(training_file, mode='rb') as f:
    train = pickle.load(f)
with open(testing_file, mode='rb') as f:
    test = pickle.load(f)
    
X_train, y_train = train['features'], train['labels']
X_test, y_test = test['features'], test['labels']

Step 1: Dataset Summary & Exploration

The pickled data is a dictionary with 4 key/value pairs:

  • 'features' is a 4D array containing raw pixel data of the traffic sign images, (num examples, width, height, channels).
  • 'labels' is a 2D array containing the label/class id of the traffic sign. The file signnames.csv contains id -> name mappings for each id.
  • 'sizes' is a list containing tuples, (width, height) representing the the original width and height the image.
  • 'coords' is a list containing tuples, (x1, y1, x2, y2) representing coordinates of a bounding box around the sign in the image. THESE COORDINATES ASSUME THE ORIGINAL IMAGE. THE PICKLED DATA CONTAINS RESIZED VERSIONS (32 by 32) OF THESE IMAGES

Complete the basic data summary below.

In [129]:
# sure to want numpy at some point here...
import numpy as np
In [3]:
## Replace each question mark with the appropriate value.

# TODO: Number of training examples
n_train = len(X_train)

# TODO: Number of testing examples.
n_test = len(X_test)

# TODO: What's the shape of an traffic sign image?
image_shape = (len(X_train[0]), len(X_train[0][0]), len(X_train[0][0][0]))
# The above makes the assumption that all images in the training data are the same size.
# This is a reasonable assumption, as this should be taken care of when preparing the training data

# TODO: How many unique classes/labels there are in the dataset.
n_classes = len(set(y_train))
# The above makes the assumption that the training set contains examples of each class
# This is a reasonable assumption, as there wouldn't be any possibility of finding any other classes!

print("Number of training examples =", n_train)
print("Number of testing examples =", n_test)
print("Image data shape =", image_shape)
print("Number of classes =", n_classes)
Number of training examples = 39209
Number of testing examples = 12630
Image data shape = (32, 32, 3)
Number of classes = 43

Visualize the German Traffic Signs Dataset using the pickled file(s). This is open ended, suggestions include: plotting traffic sign images, plotting the count of each sign, etc.

The Matplotlib examples and gallery pages are a great resource for doing visualizations in Python.

NOTE: It's recommended you start with something simple first. If you wish to do more, come back to it after you've completed the rest of the sections.

In [4]:
import random
import matplotlib.pyplot as plt
%matplotlib inline

# Let's take a look at some of the first images (10-20)

for i in range(10,20):
    image = X_train[i]
    fig = plt.figure()
    fig.suptitle('Image: ' + str(i) + ', Label: ' + str(y_train[i]), fontsize=20)
    plt.imshow(image)

# Now let's take a look at an example of each label
i = 0
while i < n_classes:
    n = 0
    while n < n_train:
        # find the first image which matches the i-th class
        if y_train[n] == i:
            # found one
            image = X_train[n]
            fig = plt.figure()
            fig.suptitle('Image: ' + str(n) + ', Label: ' + str(y_train[n]), fontsize=20)
            plt.imshow(image)
            i += 1
            n = n_train
        else:
            n += 1
/home/neil/anaconda3/envs/CarND-Traffic-Sign-Classifier-Project/lib/python3.5/site-packages/matplotlib/pyplot.py:524: RuntimeWarning: More than 20 figures have been opened. Figures created through the pyplot interface (`matplotlib.pyplot.figure`) are retained until explicitly closed and may consume too much memory. (To control this warning, see the rcParam `figure.max_open_warning`).
  max_open_warning, RuntimeWarning)
In [6]:
# encapsulate the counting/order checking in a function 
def nm_count_check_order(training_images, training_classes, print_counts=True):
    class_count = len(set(training_classes))
    image_count = len(training_images)
    counts = []
    for i in range(0, class_count):
        n_class_count = list(training_classes).count(i)
        n_first_occurrence = 0
        n_last_occurrence = len(training_images)
        n = 0
        while n < image_count:
            if training_classes[n] == i:
                n_first_occurrence = n
                n = image_count
            else:
                n += 1
        n = image_count - 1
        while n >= 0:
            if training_classes[n] == i:
                n_last_occurrence = n
                n = -1
            else:
                n -= 1
        counts += [n_class_count]
        if print_counts:
            print('Class: ' + str(i) + ', count: ' + str(n_class_count) + ', first : ' + str(n_first_occurrence) + ', last: ' + str(n_last_occurrence))
    return counts

# run the count checking
counts = nm_count_check_order(X_train, y_train, False)
print('Counts of each class: ', counts)
Counts of each class:  [210, 2220, 2250, 1410, 1980, 1860, 420, 1440, 1410, 1470, 2010, 1320, 2100, 2160, 780, 630, 420, 1110, 1200, 210, 360, 330, 390, 510, 270, 1500, 600, 240, 540, 270, 450, 780, 240, 689, 420, 1200, 390, 210, 2070, 300, 360, 240, 240]
In [7]:
N = len(counts)
x = range(N)

fig = plt.figure()
ax = plt.subplot()
plt.bar(x, counts)
ax.set_ylabel('Class Count')
ax.set_xlabel('Class Number')
ax.set_title('Class Counts in Training Images')
Out[7]:
<matplotlib.text.Text at 0x7f1226bc0470>

Notes on data exploration and findings

The images are provided in an order of ascending classification - will need to shuffle for training, which I expect will be done when splitting out a validation set prior to training.

There is a large variation in the counts for each class. Some classes have >2000 examples, some classes don't have all that many examples (e.g. 210 for class 37, 240 for class 42). Does this affect model accuracy and would a larger training set be required for a real-world self-driving car?

In terms of this modelling exercise, there are 2 aspects to the training set being unbalanced. First, the relative frequency of examples may be a reflection of the incidence of signs in the real world. If this is the case, preserving the relative frequency of the examples should cause the model to favour the more frequent signs in predictions - this would be a reflection of the prior probability of any one sign being one of the more frequent types. (E.g. if one class of sign in the world is 90% of all signs, a model which simply always guessed that sign would have a 90% chance of being correct).

Alternatively, the relative frequencies may not reflect the real world precisely and using the training set 'as-is' would therefore disadvantage those classes which appear infrequently. And classes with few training examples will not train well. This can be addressed by applying some affine translations to images in the classes with few examples - e.g. a selection of rotations or small translations, which will generate new examples for those classes. These new images can then be added into the training set to augment the training data for classes with a low number of examples.

With regard to image quality, there is a variety of levels of brightness - some images very bright, some very dark. It may aid model building if these inputs were to be normalised in a pre-processing step.

Finally, are there really only 43 types of road sign in Germany? The United Kingdom Highway Code has 100's...


Step 2: Design and Test a Model Architecture

Design and implement a deep learning model that learns to recognize traffic signs. Train and test your model on the German Traffic Sign Dataset.

There are various aspects to consider when thinking about this problem:

  • Neural network architecture
  • Play around preprocessing techniques (normalization, rgb to grayscale, etc)
  • Number of examples per label (some have more than others).
  • Generate fake data.

Here is an example of a published baseline model on this problem. It's not required to be familiar with the approach used in the paper but, it's good practice to try to read papers like these.

NOTE: The LeNet-5 implementation shown in the classroom at the end of the CNN lesson is a solid starting point. You'll have to change the number of classes and possibly the preprocessing, but aside from that it's plug and play!

Implementation

Use the code cell (or multiple code cells, if necessary) to implement the first step of your project. Once you have completed your implementation and are satisfied with the results, be sure to thoroughly answer the questions that follow.

Helper functions

The exploration suggests that some pre-processing will be required for image brightness and some additional images will need to be created.

The following cells add (and test) helper functions which create new images (via small rotations) and use the skimage library to equalise image brightness.

Functions:

rotate_image : rotate an input image

equalise_image : equalise the image levels

In [8]:
# The set of images is quite unbalanced
# To create more balance, we will add some more images
# These images will be drawn from the existing set, but made slightly different by a small rotation

# Helper function to provide image rotation - note that we want a 32x32 image to be returned in each case
from scipy import ndimage

n_height = 32
n_width  = 32
def rotate_image(in_image, degree=5, n_height=32, n_width=32):
    # rotate an image and return at the size defined by n_height x n_width
    # do the resize by trimming, as the rotate is going to add black pixels - *don't* want to use a rescale operation
    rotated = ndimage.rotate(in_image, degree)
    n_new_width = len(rotated[0])
    n_new_height = len(rotated[1])
    if n_new_width > n_width:
        n_start_width = int((n_new_width - n_width) / 2)
    else:
        n_start_width = 0
    if n_new_height > n_height:
        n_start_height = int((n_new_height - n_height) / 2)
    else:
        n_start_height = 0
    rotated = rotated[n_start_width:n_start_width+n_width, n_start_height:n_start_height+n_height]
    return rotated

# quick test of rotating and returning a new image
new_image = rotate_image(X_train[1000])
plt.imshow(new_image)
print('Image dimensions :', len(new_image[0]), ',', len(new_image[1]))
Image dimensions : 32 , 32
In [9]:
# image intensity equalisation - can use skimage.exposure.equalize_hist()
# Note that using this library needs a lot of faffing about with the Anaconda installation
# There are some missing links to the MKL build in Anaconda, which messes up skimage
# See http://stackoverflow.com/questions/36659453/intel-mkl-fatal-error-cannot-load-libmkl-avx2-so-or-libmkl-def-so 

from skimage import exposure
def equalise_image(i):
    return exposure.equalize_hist(i)

# Test - use an already good image
plt.figure()
plt.imshow(X_train[1000])
# Equalise the image colours
plt.figure()
img_eq = equalise_image(X_train[1000]) 
plt.imshow(img_eq)

# Test - this first image is very dark
plt.figure()
plt.imshow(X_train[2430])
# Equalise the image colours
plt.figure()
img_eq = equalise_image(X_train[2430]) 
plt.imshow(img_eq)

# Test - this first image is va blue sign
plt.figure()
plt.imshow(X_train[33090])
# Equalise the image colours
plt.figure()
img_eq = equalise_image(X_train[33090]) 
plt.imshow(img_eq)
/home/neil/anaconda3/envs/CarND-Traffic-Sign-Classifier-Project/lib/python3.5/site-packages/skimage/exposure/exposure.py:63: UserWarning: This might be a color image. The histogram will be computed on the flattened image. You can instead apply this function to each color channel.
  warn("This might be a color image. The histogram will be "
Out[9]:
<matplotlib.image.AxesImage at 0x7f11fc03aa58>
In [10]:
# Balance out the training data
# This code creates additional images for the sparse classes, using small rotations
# We want any class to have at least 1/3 of the number of samples for the largest

def create_extra_images(training_images, training_labels):
    # Find the max count over all classes
    num_classes = len(set(training_labels))
    n_max_class_sample_size = 0 
    for i in range(0, num_classes):
        n_class_count = list(training_labels).count(i)
        if n_class_count > n_max_class_sample_size:
            n_max_class_sample_size = n_class_count
    print('Max count across all classes :', n_max_class_sample_size)

    # For each class with less than the max count of images, add some more images
    # Rotate by +/-1, +/-5, +/-10 degrees
    fFirst = True
    for i in range(0, num_classes):
        print('Processing class :', i)
        n_class_count = list(training_labels).count(i)
        n_max_to_add = n_max_class_sample_size / 3               # maximum images to add
        n_max_to_add -= n_class_count                            # less the images already present for this class
        n_index = 0
        while (n_index < len(training_images)) and (n_max_to_add > 0):
            # create some new images
            if training_labels[n_index] == i:
                # this is an image for the class we want to add into
                for degree in [1,-1,5,-5,10,-10]:
                    new_image = rotate_image(training_images[n_index],degree) # create a rotated image, assume 32x32 sizing
                    if fFirst:
                        new_images = [new_image]
                        new_labels = [i]
                        fFirst = False
                    else:
                        new_images = np.append(new_images, [new_image], axis=0)
                        new_labels = np.append(new_labels, [i], axis = 0)
                    n_max_to_add -= 1
                n_index += 1                                          # move the next image, ready to replicate that
            else:
                n_index += 1
        if (n_max_class_sample_size / 3) - n_class_count > 0:
            print('Created ', int((n_max_class_sample_size / 3) - n_class_count), ' new images')
        else:
            print('No additional images created')

    if not fFirst:
        print('Extra images created :', len(new_images))
    else:
        print('No extra images created?')
        
    return new_images, new_labels

new_images, new_labels = create_extra_images(X_train, y_train)
Max count across all classes : 2250
Processing class : 0
Created  540  new images
Processing class : 1
No additional images created
Processing class : 2
No additional images created
Processing class : 3
No additional images created
Processing class : 4
No additional images created
Processing class : 5
No additional images created
Processing class : 6
Created  330  new images
Processing class : 7
No additional images created
Processing class : 8
No additional images created
Processing class : 9
No additional images created
Processing class : 10
No additional images created
Processing class : 11
No additional images created
Processing class : 12
No additional images created
Processing class : 13
No additional images created
Processing class : 14
No additional images created
Processing class : 15
Created  120  new images
Processing class : 16
Created  330  new images
Processing class : 17
No additional images created
Processing class : 18
No additional images created
Processing class : 19
Created  540  new images
Processing class : 20
Created  390  new images
Processing class : 21
Created  420  new images
Processing class : 22
Created  360  new images
Processing class : 23
Created  240  new images
Processing class : 24
Created  480  new images
Processing class : 25
No additional images created
Processing class : 26
Created  150  new images
Processing class : 27
Created  510  new images
Processing class : 28
Created  210  new images
Processing class : 29
Created  480  new images
Processing class : 30
Created  300  new images
Processing class : 31
No additional images created
Processing class : 32
Created  510  new images
Processing class : 33
Created  61  new images
Processing class : 34
Created  330  new images
Processing class : 35
No additional images created
Processing class : 36
Created  360  new images
Processing class : 37
Created  540  new images
Processing class : 38
No additional images created
Processing class : 39
Created  450  new images
Processing class : 40
Created  390  new images
Processing class : 41
Created  510  new images
Processing class : 42
Created  510  new images
Extra images created : 9066
In [11]:
print(new_labels)
print(len(new_labels))
[ 0  0  0 ..., 42 42 42]
9066
In [12]:
# Visualise what the concatenation of X_train and new_images is going to look like

conc_images = np.append(X_train, new_images, axis=0)
conc_labels = np.append(y_train, new_labels, axis=0)

counts = nm_count_check_order(conc_images, conc_labels, False)
print('Counts of each augmented class: ', counts)

N = len(counts)
x = range(N)

fig = plt.figure()
ax = plt.subplot()
plt.bar(x, counts)
ax.set_ylabel('Class Count')
ax.set_xlabel('Class Number')
ax.set_title('Class Counts in Augmented Training Images')
Counts of each augmented class:  [750, 2220, 2250, 1410, 1980, 1860, 750, 1440, 1410, 1470, 2010, 1320, 2100, 2160, 780, 750, 750, 1110, 1200, 750, 750, 750, 750, 750, 750, 1500, 750, 750, 750, 750, 750, 780, 750, 755, 750, 1200, 750, 750, 2070, 750, 750, 750, 750]
Out[12]:
<matplotlib.text.Text at 0x7f11fbfa8c88>

Now want to create some alternative sets of training data (to use for comparative models)

Set #1: The original X_train, y_train sets

Set #2: The original X_train, y_train sets, with pre-processing to balance the brightness

Set #3: The original X_train, y_train sets, augmented with additional images for the sparse classes

Set #4: The augmented set (additional images), with image brightness balancing

These options will allow the network to be trained on each and the validation accuracies of each run compared.

Note that when augmenting the data with additional images, we want to take a validation set from the original set of images before creating the augmentation. This will avoid running validations on copies of images which are in the training set.

In [13]:
# Helper function to split out some training and validation data
# Note that this is just a slice at this stage and isn't yet randomising the order of the training samples
# Reason for putting this in place early is that some of the sets will take validation data before augmenting
# the training set - to avoid having near-duplicate images in the training and validation sets.

def train_validation_split(image_array, label_array, train_percent=0.8):
    n = int(len(image_array) * train_percent)
    train_i = image_array[:n]
    valid_i = image_array[n:]
    train_l = label_array[:n]
    valid_l = label_array[n:]
    return train_i, train_l, valid_i, valid_l
In [14]:
### Preprocess the data here.
### Feel free to use as many code cells as needed.

# At this stage have inputs of:
# X_train - initial input images
# y_train - labels for those initial images
# Both sets neatly ordered by class label (ascending)

from sklearn.utils import shuffle

# Set #1: as given
training_images_1, training_labels_1 = shuffle(X_train, y_train)
training_images_1, training_labels_1, validation_images_1, validation_labels_1 = train_validation_split(training_images_1, training_labels_1, 0.9)
print('Size of basic training set   : ', len(training_images_1))
print('Size of basic validation set : ', len(validation_images_1))

# Set #2: apply the brightness function to all of the images in the first set
# using the already-shuffled set means that the validation set will be the same - better comparison
training_images_2 = []
training_labels_2 = []
validation_images_2 = []
validation_labels_2 = []
for idx in range(len(training_images_1)):
    training_images_2.append(equalise_image(training_images_1[idx]))
    training_labels_2.append(training_labels_1[idx])
print('Size of brightness-adjusted training set: ', len(training_images_2))
for idx in range(len(validation_images_1)):
    validation_images_2.append(equalise_image(validation_images_1[idx]))
    validation_labels_2.append(validation_labels_1[idx])
print('Size of brightness-adjusted validation set: ', len(validation_images_2))
      
# Set #3: use the existing set of validation data, then append on some additional images to the training set
# First create some extra images, from the training set for set #1
new_images_3, new_labels_3 = create_extra_images(training_images_1, training_labels_1)
training_images_3 = np.append(training_images_1, new_images_3, axis=0)
training_labels_3 = np.append(training_labels_1, new_labels_3, axis=0)
# Need to give these another shuffle
training_images_3, training_labels_3 = shuffle(training_images_3, training_labels_3)
# Can just use a copy of the set #1 validation images
validation_images_3 = validation_images_1
validation_labels_3 = validation_labels_1
print('Size of augmented training set   : ', len(training_images_3))
print('Size of augmented validation set : ', len(validation_images_3))

# Set #4: apply the brightness equalisation function to all of the images in the third set
# All of the images are already shuffled and the set augmented etc
training_images_4 = []
training_labels_4 = []
validation_images_4 = []
validation_labels_4 = []
for idx in range(len(training_images_3)):
      training_images_4.append(equalise_image(training_images_3[idx]))
      training_labels_4.append(training_labels_3[idx])
print('Size of brightness-adjusted and augmented training set: ', len(training_images_4))
for idx in range(len(validation_images_3)):
      validation_images_4.append(equalise_image(validation_images_3[idx]))
      validation_labels_4.append(validation_labels_3[idx])
print('Size of brightness-adjusted and augmented validation set: ', len(validation_images_4))
Size of basic training set   :  35288
Size of basic validation set :  3921
/home/neil/anaconda3/envs/CarND-Traffic-Sign-Classifier-Project/lib/python3.5/site-packages/skimage/exposure/exposure.py:63: UserWarning: This might be a color image. The histogram will be computed on the flattened image. You can instead apply this function to each color channel.
  warn("This might be a color image. The histogram will be "
Size of brightness-adjusted training set:  35288
Size of brightness-adjusted validation set:  3921
Max count across all classes : 2000
Processing class : 0
Created  481  new images
Processing class : 1
No additional images created
Processing class : 2
No additional images created
Processing class : 3
No additional images created
Processing class : 4
No additional images created
Processing class : 5
No additional images created
Processing class : 6
Created  287  new images
Processing class : 7
No additional images created
Processing class : 8
No additional images created
Processing class : 9
No additional images created
Processing class : 10
No additional images created
Processing class : 11
No additional images created
Processing class : 12
No additional images created
Processing class : 13
No additional images created
Processing class : 14
No additional images created
Processing class : 15
Created  112  new images
Processing class : 16
Created  280  new images
Processing class : 17
No additional images created
Processing class : 18
No additional images created
Processing class : 19
Created  476  new images
Processing class : 20
Created  341  new images
Processing class : 21
Created  362  new images
Processing class : 22
Created  309  new images
Processing class : 23
Created  215  new images
Processing class : 24
Created  425  new images
Processing class : 25
No additional images created
Processing class : 26
Created  132  new images
Processing class : 27
Created  453  new images
Processing class : 28
Created  186  new images
Processing class : 29
Created  422  new images
Processing class : 30
Created  263  new images
Processing class : 31
No additional images created
Processing class : 32
Created  456  new images
Processing class : 33
Created  46  new images
Processing class : 34
Created  290  new images
Processing class : 35
No additional images created
Processing class : 36
Created  318  new images
Processing class : 37
Created  471  new images
Processing class : 38
No additional images created
Processing class : 39
Created  400  new images
Processing class : 40
Created  339  new images
Processing class : 41
Created  447  new images
Processing class : 42
Created  448  new images
Extra images created : 8034
Size of augmented training set   :  43322
Size of augmented validation set :  3921
Size of brightness-adjusted and augmented training set:  43322
Size of brightness-adjusted and augmented validation set:  3921
In [15]:
# Check that the training data is now shuffled
# Should not see an increasing first/last image sequence over the classes any more

nm_count_check_order(training_images_1, training_labels_1)
Class: 0, count: 185, first : 203, last: 35057
Class: 1, count: 2000, first : 31, last: 35285
Class: 2, count: 1991, first : 1, last: 35249
Class: 3, count: 1278, first : 8, last: 35225
Class: 4, count: 1785, first : 10, last: 35264
Class: 5, count: 1653, first : 12, last: 35262
Class: 6, count: 379, first : 70, last: 35273
Class: 7, count: 1279, first : 37, last: 35281
Class: 8, count: 1272, first : 26, last: 35242
Class: 9, count: 1313, first : 5, last: 35279
Class: 10, count: 1845, first : 3, last: 35179
Class: 11, count: 1194, first : 72, last: 35232
Class: 12, count: 1888, first : 36, last: 35283
Class: 13, count: 1947, first : 0, last: 35287
Class: 14, count: 714, first : 47, last: 35154
Class: 15, count: 554, first : 27, last: 35282
Class: 16, count: 386, first : 100, last: 35259
Class: 17, count: 995, first : 113, last: 35261
Class: 18, count: 1083, first : 29, last: 35255
Class: 19, count: 190, first : 167, last: 35199
Class: 20, count: 325, first : 17, last: 35201
Class: 21, count: 304, first : 57, last: 35189
Class: 22, count: 357, first : 88, last: 35180
Class: 23, count: 451, first : 16, last: 35231
Class: 24, count: 241, first : 51, last: 35211
Class: 25, count: 1354, first : 23, last: 35263
Class: 26, count: 534, first : 94, last: 35192
Class: 27, count: 213, first : 422, last: 34958
Class: 28, count: 480, first : 46, last: 35286
Class: 29, count: 244, first : 40, last: 35247
Class: 30, count: 403, first : 144, last: 35274
Class: 31, count: 705, first : 7, last: 35212
Class: 32, count: 210, first : 178, last: 35124
Class: 33, count: 620, first : 30, last: 35236
Class: 34, count: 376, first : 48, last: 34993
Class: 35, count: 1090, first : 20, last: 35245
Class: 36, count: 348, first : 350, last: 35228
Class: 37, count: 195, first : 75, last: 34786
Class: 38, count: 1877, first : 6, last: 35272
Class: 39, count: 266, first : 15, last: 34979
Class: 40, count: 327, first : 54, last: 35284
Class: 41, count: 219, first : 97, last: 35248
Class: 42, count: 218, first : 2, last: 35074
Out[15]:
[185,
 2000,
 1991,
 1278,
 1785,
 1653,
 379,
 1279,
 1272,
 1313,
 1845,
 1194,
 1888,
 1947,
 714,
 554,
 386,
 995,
 1083,
 190,
 325,
 304,
 357,
 451,
 241,
 1354,
 534,
 213,
 480,
 244,
 403,
 705,
 210,
 620,
 376,
 1090,
 348,
 195,
 1877,
 266,
 327,
 219,
 218]

Question 1

Describe how you preprocessed the data. Why did you choose that technique?

Answer: My aim is to start with the LeNet example architecture, which requires the data to be in a certain format.

The images are in a 32x32x3 format. 32x32 is the image size which the example LeNet process accepts and I'm planning to use that architecture as at least the starting point for my model - so there's nothing to do in terms of changing the size of images (i.e. no need for any padding).

LeNet uses depth 1 images (greyscale), but colour is important for traffic sign classification, so I'm going to keep this at depth 3.

However, there are 2 points arising from the investigation into the supplied training data - the numbers of example images in each class of sign and the contrast/brightness of the images.

First, the low numbers of images in some classes are a cause for concern. This may be useful information in terms of the relative probabilities of each sign, but may also be an issue for training the model (insufficient samples for the sparse classes). To address this, it is possible to create additional examples images via transformations of some existing images (affine transformations, such as rotation).

Second, the training images are provided with a range of brightnesses - potentially reflecting differing lighting conditions under which the images were captured. This can be addressed by equalisation of the lighting levels within each image.

Clearly, there is the option to combine these pre-processing actions. As I don't know whether the pre-processing will positively or negatively impact the model predictive accuracy, I have chosen to provide a training and validation set for each of the combinations:

Set 1: the training data, as provided.

Set 2: the training data, with image light levels balanced

Set 3: the training data, augmented with additional images created from the samples using small rotations

Set 4: the training data both augmented and with image brightness processing

Creating a model on each of these sets of training data and comparing validation accuracies, will allow the effect of each pre-processing step to be judged.

Note: to ensure that the comparisons are fair, the same validation set has been extracted - including extraction prior to the creation of additional images. This ensures that there is no risk of validating with images which are only rotated copies of images used for training.

Where the brightness equalisation transformation has been applied to training data, this transformation has also been applied to the validation set.

In [17]:
### Generate data additional data (OPTIONAL!)
### and split the data into training/validation/testing sets here.
### Feel free to use as many code cells as needed.

# Four sets of training/validation data have already been prepared, as per the discussions above

print('Set #1: basic training set - as provided')
print('Training images count   : ', len(training_images_1))
print('Validation images count : ', len(validation_images_1))
print()
print('Set #2: basic training set, with equalised image brightnesses')
print('Training images count   : ', len(training_images_2))
print('Validation images count : ', len(validation_images_2))
print()
print('Set #3: basic training set, with additional images')
print('Training images count   : ', len(training_images_3))
print('Validation images count : ', len(validation_images_3))
print()
print('Set #4: augmented set, with equalised image brightnesses')
print('Training images count   : ', len(training_images_4))
print('Validation images count : ', len(validation_images_4))
print()
Set #1: basic training set - as provided
Training images count   :  35288
Validation images count :  3921

Set #2: basic training set, with equalised image brightnesses
Training images count   :  35288
Validation images count :  3921

Set #3: basic training set, with additional images
Training images count   :  43322
Validation images count :  3921

Set #4: augmented set, with equalised image brightnesses
Training images count   :  43322
Validation images count :  3921

Question 2

Describe how you set up the training, validation and testing data for your model. Optional: If you generated additional data, how did you generate the data? Why did you generate the data? What are the differences in the new dataset (with generated data) from the original dataset?

Answer: The setup of training and validation data is simply a process to split the X_train set of images into a proportion for training and a proportion for model validation. This is done with a simple slice operation, once the images have been shuffled into random order. Given the sizes of the data set, I've used a 90/10 split between training and validation data. This looks to give a good size of validation set (3921) for checking of the model, whilst maximising the amount of data available for training (a common split is 80/20 training/validation - I've chosen to keep a little more data for training).

Note that the lesson videos suggest using sklearn.model_selection.train_test_split - I've simply chosed to achieve the same result with a slice and avoid another random shuffling of the data (which would be fine, but with two arrays I'm always concerned about the labels becoming detached from the images). This has the added benefit that with a single shuffle operation, I can be assured that the same validation set is being used for each of the 4 test sets.

There is no need to create a test set from X_train - this has already been done and test set has been provided in the original pickle input files.

However, for final testing of models which use the image brightness equalisation, the test set should have a similar transformation applied as a pre-processing step.

Some additional data has been created for 2 of the sample sets, to increase the number of samples in the sparsely populated classes of sign. This has been done with small rotations, simply to provide more examples for the learning process. The difference between these augmented sets and the original is the addition of c.8,000 additional images to the sparse classes. As described previously, care has been taken that the validation sets do not contain duplicates of the additional images used for training.

In [18]:
# Set up tensorflow
# Various global hyper-parameters to be defined here
import tensorflow as tf
from tensorflow.contrib.layers import flatten        # used for the flatten layer in the network setup

EPOCHS = 50         # Fairly standard initial setting, balancing accuracy with the length of time to train
BATCH_SIZE = 128    # Larger batches mean faster training, but may be limited by memory
mu = 0              # mean value of distribution used for random assignment of initial weights
sigma = 0.1         # standard deviation used for random assignment of initial weights - want this to be small
rate = 0.001        # learning rate, 0.001 is a good initial setting for a LeNet-type network
In [19]:
# Set up the network operations
# The input X is going to be the batch of training images
def SignIdentificationNetwork(X):
    
    # First convolutional layer - output 28x28x6 (further reduced by pooling step)
    conv1_W = tf.Variable(tf.truncated_normal(shape=(5,5,3,6),mean=mu,stddev=sigma))
    conv1_b = tf.Variable(tf.zeros(6))
    conv1   = tf.nn.conv2d(X, conv1_W, strides=[1,1,1,1], padding='VALID') + conv1_b
    # Activation - relu function
    conv1   = tf.nn.relu(conv1)   
    # Pooling step - get down to 14x14x6 output, using max_pool with 2x2 kernels
    conv1   = tf.nn.max_pool(conv1, ksize=[1,2,2,1], strides=[1,2,2,1], padding='VALID')  
    
    # Second convolutional layer - output 10x10x16 (further reduced by pooling step)
    conv2_W = tf.Variable(tf.truncated_normal(shape=(5,5,6,16), mean=mu, stddev=sigma))
    conv2_b = tf.Variable(tf.zeros(16))
    conv2   = tf.nn.conv2d(conv1, conv2_W, strides=[1,1,1,1], padding='VALID') + conv2_b
    # Activation - relu again
    conv2   = tf.nn.relu(conv2)
    # Pooling step - get down to 5x5x16 output
    conv2   = tf.nn.max_pool(conv2, ksize=[1,2,2,1], strides=[1,2,2,1], padding='VALID')
    
    # Flatten step - go from 5x5x16 input to 1x400 output
    flat1   = flatten(conv2)
    
    # First fully connected layer - input 400, output 120
    full1_W = tf.Variable(tf.truncated_normal(shape=(400,120), mean=mu, stddev=sigma))
    full1_b = tf.Variable(tf.zeros(120))
    full1   = tf.matmul(flat1, full1_W) + full1_b
    # Activation - relu again
    full1   = tf.nn.relu(full1)
    
    # Second fully connected layer - input 120, output 84
    full2_W = tf.Variable(tf.truncated_normal(shape=(120,84), mean=mu, stddev=sigma))
    full2_b = tf.Variable(tf.zeros(84))
    full2   = tf.matmul(full1, full2_W) + full2_b
    # Activation - relu again
    full2   = tf.nn.relu(full2)
    
    # Third fully connected layer - input 84, output 43 (number of classes required)
    full3_W = tf.Variable(tf.truncated_normal(shape=(84,43), mean=mu, stddev=sigma))
    full3_b = tf.Variable(tf.zeros(43))
    logits  = tf.matmul(full2, full3_W) + full3_b
    
    return logits
In [20]:
# Set up placeholders
X = tf.placeholder(tf.float32, (None,32,32,3))      # dimensions set for 32x32x3 images
y = tf.placeholder(tf.int32, (None))
one_hot_y = tf.one_hot(y, 43)                       # sized for 43 output classes
In [21]:
# Set up the training pipeline and optimiser
logits = SignIdentificationNetwork(X)
cross_entropy = tf.nn.softmax_cross_entropy_with_logits(logits, one_hot_y)
loss_operation = tf.reduce_mean(cross_entropy)
optimizer = tf.train.AdamOptimizer(learning_rate = rate)      # using the Adam optimiser - build in, better than SGD
training_operation = optimizer.minimize(loss_operation)
In [22]:
# Set up the evaluation function and related operations

# correct prediction if prediction matches the one-hot value
correct_prediction = tf.equal(tf.argmax(logits, 1), tf.argmax(one_hot_y, 1))
accuracy_operation = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))

# standard evaluation function
def evaluate(X_data, y_data):
    num_examples = len(X_data)
    total_accuracy = 0
    sess = tf.get_default_session()
    for offset in range(0, num_examples, BATCH_SIZE):
        batch_x, batch_y = X_data[offset:offset+BATCH_SIZE], y_data[offset:offset+BATCH_SIZE]
        accuracy = sess.run(accuracy_operation, feed_dict={X: batch_x, y: batch_y})
        total_accuracy += (accuracy * len(batch_x))
    return total_accuracy / num_examples

Question 3

What does your final architecture look like? (Type of model, layers, sizes, connectivity, etc.) For reference on how to build a deep neural network using TensorFlow, see Deep Neural Network in TensorFlow from the classroom.

Answer:

The architecture above is a slight modification of the LeNet architecture used for recognition of handwritten digits. The modifications are to allow inputs of colour images and to allow for the wider output set.

LeNet architecture

The LeNet architecture has the following layers:

1: First convolutional layer - down from 32x32x3 to 28x28x6, with RELU activation followed by pooling to 14x14x6.

2: Second convolutional layer - down to 10x10x16, with RELU activation followed by more pooling down to 5x5x16. LeNet also has a flatten at this point, down to a single colour channel.

3: First fully connected layer - with 120 outputs.

4: Second fully connected layer - with 84 outputs.

5: Third fully connected layer - for digit-identification LeNet this is down to 10 outputs (0-9), whilst our sign identification requires 43 outputs (one for each sign class)

In [23]:
# Some time functions - useful to keep sane when waiting for the training to complete
import time
In [24]:
### Train your model here.
### Feel free to use as many code cells as needed.

# Train the model with the first training set and save out the session data

start_time = time.time()

with tf.Session() as sess:
    sess.run(tf.global_variables_initializer())
    num_examples = len(training_images_1)
    
    print('Training started, time to grab a cuppa...')
    print()
    print('Training for set #1: basic training images')
    print('Set size      : ', num_examples)
    print('Epochs        : ', EPOCHS)
    print('Batch size    : ', BATCH_SIZE)
    print('Learning rate : ', rate)
    print()
    
    for i in range(EPOCHS):
        start_epoch = time.time()
        # take a shuffled copy of the training data for each epoch
        tr_images, tr_labels = shuffle(training_images_1, training_labels_1)
        for offset in range(i, num_examples, BATCH_SIZE):
            end = offset + BATCH_SIZE
            batch_x, batch_y = tr_images[offset:end], tr_labels[offset:end]
            sess.run(training_operation, feed_dict={X: batch_x, y: batch_y})
            
        validation_accuracy = evaluate(validation_images_1, validation_labels_1)
        print('EPOCH               :', i+1, ' of ', EPOCHS)
        print('Validation accuracy :', validation_accuracy)
        print()
        
        end_epoch = time.time()
        print('Epoch processing time (secs)    : ', end_epoch - start_epoch)
        print('Total elapsed time (mins)       : ', (end_epoch - start_time)/60.0)
        print('Estimated time remaining (mins) : ', (end_epoch - start_epoch) * (EPOCHS - (i+1)) / 60.0)
        print()
        
    # save the model
    try:
        saver
    except NameError:
        saver = tf.train.Saver()
    saver.save(sess, 'signs_set1')
    print('Saved model as *signs_set1*')
Training started, time to grab a cuppa...

Training for set #1: basic training images
Set size      :  35288
Epochs        :  50
Batch size    :  128
Learning rate :  0.001

EPOCH               : 1  of  50
Validation accuracy : 0.682223922727

Epoch processing time (secs)    :  17.887845754623413
Total elapsed time (mins)       :  0.30075364112854003
Estimated time remaining (mins) :  14.608407366275788

EPOCH               : 2  of  50
Validation accuracy : 0.851823514227

Epoch processing time (secs)    :  19.25080680847168
Total elapsed time (mins)       :  0.6216228087743123
Estimated time remaining (mins) :  15.400645446777343

EPOCH               : 3  of  50
Validation accuracy : 0.90359602156

Epoch processing time (secs)    :  17.161314249038696
Total elapsed time (mins)       :  0.9076705614725749
Estimated time remaining (mins) :  13.443029495080312

EPOCH               : 4  of  50
Validation accuracy : 0.919663350867

Epoch processing time (secs)    :  17.670385360717773
Total elapsed time (mins)       :  1.2022079388300577
Estimated time remaining (mins) :  13.54729544321696

EPOCH               : 5  of  50
Validation accuracy : 0.938281050433

Epoch processing time (secs)    :  19.187076568603516
Total elapsed time (mins)       :  1.5220203797022502
Estimated time remaining (mins) :  14.390307426452637

EPOCH               : 6  of  50
Validation accuracy : 0.947462382501

Epoch processing time (secs)    :  19.94297957420349
Total elapsed time (mins)       :  1.8544358293215433
Estimated time remaining (mins) :  14.624851687749226

EPOCH               : 7  of  50
Validation accuracy : 0.94720734505

Epoch processing time (secs)    :  18.365164041519165
Total elapsed time (mins)       :  2.160547188917796
Estimated time remaining (mins) :  13.161700896422069

EPOCH               : 8  of  50
Validation accuracy : 0.956898750319

Epoch processing time (secs)    :  18.269646883010864
Total elapsed time (mins)       :  2.465062344074249
Estimated time remaining (mins) :  12.788752818107605

EPOCH               : 9  of  50
Validation accuracy : 0.965570007651

Epoch processing time (secs)    :  19.380022525787354
Total elapsed time (mins)       :  2.7880855162938434
Estimated time remaining (mins) :  13.243015392621357

EPOCH               : 10  of  50
Validation accuracy : 0.969905636469

Epoch processing time (secs)    :  17.958415746688843
Total elapsed time (mins)       :  3.087419613202413
Estimated time remaining (mins) :  11.972277164459229

EPOCH               : 11  of  50
Validation accuracy : 0.964549859882

Epoch processing time (secs)    :  18.470076322555542
Total elapsed time (mins)       :  3.3952768484751386
Estimated time remaining (mins) :  12.005549609661102

EPOCH               : 12  of  50
Validation accuracy : 0.97194593216

Epoch processing time (secs)    :  18.741915702819824
Total elapsed time (mins)       :  3.7076665997505187
Estimated time remaining (mins) :  11.869879945119221

EPOCH               : 13  of  50
Validation accuracy : 0.971435858351

Epoch processing time (secs)    :  18.403620719909668
Total elapsed time (mins)       :  4.0144244511922205
Estimated time remaining (mins) :  11.348899443944296

EPOCH               : 14  of  50
Validation accuracy : 0.965059933994

Epoch processing time (secs)    :  18.335832834243774
Total elapsed time (mins)       :  4.320047601064046
Estimated time remaining (mins) :  11.001499700546265

EPOCH               : 15  of  50
Validation accuracy : 0.976791635091

Epoch processing time (secs)    :  18.71116280555725
Total elapsed time (mins)       :  4.631925288836161
Estimated time remaining (mins) :  10.914844969908396

EPOCH               : 16  of  50
Validation accuracy : 0.968120377607

Epoch processing time (secs)    :  18.377058267593384
Total elapsed time (mins)       :  4.9382336695988975
Estimated time remaining (mins) :  10.41366635163625

EPOCH               : 17  of  50
Validation accuracy : 0.973986228155

Epoch processing time (secs)    :  18.46223735809326
Total elapsed time (mins)       :  5.245962448914846
Estimated time remaining (mins) :  10.154230546951293

EPOCH               : 18  of  50
Validation accuracy : 0.967100229533

Epoch processing time (secs)    :  18.65932035446167
Total elapsed time (mins)       :  5.556977041562399
Estimated time remaining (mins) :  9.951637522379558

EPOCH               : 19  of  50
Validation accuracy : 0.972711043253

Epoch processing time (secs)    :  21.338554620742798
Total elapsed time (mins)       :  5.912642105420431
Estimated time remaining (mins) :  11.02491988738378

EPOCH               : 20  of  50
Validation accuracy : 0.978321856669

Epoch processing time (secs)    :  19.08697509765625
Total elapsed time (mins)       :  6.230784773826599
Estimated time remaining (mins) :  9.543487548828125

EPOCH               : 21  of  50
Validation accuracy : 0.964294822749

Epoch processing time (secs)    :  18.828803062438965
Total elapsed time (mins)       :  6.544626621405284
Estimated time remaining (mins) :  9.1005881468455

EPOCH               : 22  of  50
Validation accuracy : 0.970670747258

Epoch processing time (secs)    :  19.41171932220459
Total elapsed time (mins)       :  6.868181542555491
Estimated time remaining (mins) :  9.058802350362141

EPOCH               : 23  of  50
Validation accuracy : 0.967610303494

Epoch processing time (secs)    :  19.164900302886963
Total elapsed time (mins)       :  7.187621299425761
Estimated time remaining (mins) :  8.624205136299134

EPOCH               : 24  of  50
Validation accuracy : 0.972966080234

Epoch processing time (secs)    :  19.982391834259033
Total elapsed time (mins)       :  7.52069593667984
Estimated time remaining (mins) :  8.659036461512247

EPOCH               : 25  of  50
Validation accuracy : 0.981382300434

Epoch processing time (secs)    :  22.99721598625183
Total elapsed time (mins)       :  7.904010264078776
Estimated time remaining (mins) :  9.582173327604929

EPOCH               : 26  of  50
Validation accuracy : 0.974241265135

Epoch processing time (secs)    :  19.80038833618164
Total elapsed time (mins)       :  8.234030445416769
Estimated time remaining (mins) :  7.920155334472656

EPOCH               : 27  of  50
Validation accuracy : 0.97857689365

Epoch processing time (secs)    :  19.166261911392212
Total elapsed time (mins)       :  8.553495009740194
Estimated time remaining (mins) :  7.347067066033682

EPOCH               : 28  of  50
Validation accuracy : 0.96914052568

Epoch processing time (secs)    :  19.29429602622986
Total elapsed time (mins)       :  8.875094950199127
Estimated time remaining (mins) :  7.074575209617615

EPOCH               : 29  of  50
Validation accuracy : 0.975516449885

Epoch processing time (secs)    :  19.638762712478638
Total elapsed time (mins)       :  9.202432028452556
Estimated time remaining (mins) :  6.8735669493675235

EPOCH               : 30  of  50
Validation accuracy : 0.980617189492

Epoch processing time (secs)    :  20.487886428833008
Total elapsed time (mins)       :  9.543920918305714
Estimated time remaining (mins) :  6.829295476277669

EPOCH               : 31  of  50
Validation accuracy : 0.976791634787

Epoch processing time (secs)    :  21.074836492538452
Total elapsed time (mins)       :  9.895195583502451
Estimated time remaining (mins) :  6.673698222637176

EPOCH               : 32  of  50
Validation accuracy : 0.977556746032

Epoch processing time (secs)    :  19.6901216506958
Total elapsed time (mins)       :  10.223389160633086
Estimated time remaining (mins) :  5.90703649520874

EPOCH               : 33  of  50
Validation accuracy : 0.97755674588

Epoch processing time (secs)    :  19.93753457069397
Total elapsed time (mins)       :  10.555710387229919
Estimated time remaining (mins) :  5.6489681283632915

EPOCH               : 34  of  50
Validation accuracy : 0.977556746184

Epoch processing time (secs)    :  20.064331531524658
Total elapsed time (mins)       :  10.890141503016155
Estimated time remaining (mins) :  5.350488408406576

EPOCH               : 35  of  50
Validation accuracy : 0.976026523998

Epoch processing time (secs)    :  19.760637283325195
Total elapsed time (mins)       :  11.219513173898061
Estimated time remaining (mins) :  4.940159320831299

EPOCH               : 36  of  50
Validation accuracy : 0.975516450037

Epoch processing time (secs)    :  19.89067792892456
Total elapsed time (mins)       :  11.551050198078155
Estimated time remaining (mins) :  4.6411581834157305

EPOCH               : 37  of  50
Validation accuracy : 0.981382300586

Epoch processing time (secs)    :  23.134939432144165
Total elapsed time (mins)       :  11.93666004339854
Estimated time remaining (mins) :  5.012570210297903

EPOCH               : 38  of  50
Validation accuracy : 0.977556746336

Epoch processing time (secs)    :  22.4201979637146
Total elapsed time (mins)       :  12.310350867112478
Estimated time remaining (mins) :  4.48403959274292

EPOCH               : 39  of  50
Validation accuracy : 0.984697781482

Epoch processing time (secs)    :  21.44640326499939
Total elapsed time (mins)       :  12.667822329203288
Estimated time remaining (mins) :  3.9318405985832214

EPOCH               : 40  of  50
Validation accuracy : 0.978831930782

Epoch processing time (secs)    :  20.763585090637207
Total elapsed time (mins)       :  13.013910098870594
Estimated time remaining (mins) :  3.460597515106201

EPOCH               : 41  of  50
Validation accuracy : 0.975006376077

Epoch processing time (secs)    :  21.022809267044067
Total elapsed time (mins)       :  13.364313503106436
Estimated time remaining (mins) :  3.15342139005661

EPOCH               : 42  of  50
Validation accuracy : 0.98342259658

Epoch processing time (secs)    :  21.162367820739746
Total elapsed time (mins)       :  13.717046829064687
Estimated time remaining (mins) :  2.8216490427652996

EPOCH               : 43  of  50
Validation accuracy : 0.976791634939

Epoch processing time (secs)    :  20.55935525894165
Total elapsed time (mins)       :  14.05972861846288
Estimated time remaining (mins) :  2.3985914468765257

EPOCH               : 44  of  50
Validation accuracy : 0.981892374546

Epoch processing time (secs)    :  21.871447563171387
Total elapsed time (mins)       :  14.424278378486633
Estimated time remaining (mins) :  2.1871447563171387

EPOCH               : 45  of  50
Validation accuracy : 0.98469778133

Epoch processing time (secs)    :  24.611825466156006
Total elapsed time (mins)       :  14.834497813383738
Estimated time remaining (mins) :  2.0509854555130005

EPOCH               : 46  of  50
Validation accuracy : 0.973986228155

Epoch processing time (secs)    :  21.31803297996521
Total elapsed time (mins)       :  15.189822363853455
Estimated time remaining (mins) :  1.4212021986643473

EPOCH               : 47  of  50
Validation accuracy : 0.978831930934

Epoch processing time (secs)    :  21.00021719932556
Total elapsed time (mins)       :  15.539852646986644
Estimated time remaining (mins) :  1.050010859966278

EPOCH               : 48  of  50
Validation accuracy : 0.982402448355

Epoch processing time (secs)    :  23.073112726211548
Total elapsed time (mins)       :  15.924429937203724
Estimated time remaining (mins) :  0.769103757540385

EPOCH               : 49  of  50
Validation accuracy : 0.983932670389

Epoch processing time (secs)    :  21.667988538742065
Total elapsed time (mins)       :  16.285588955879213
Estimated time remaining (mins) :  0.36113314231236776

EPOCH               : 50  of  50
Validation accuracy : 0.968120376983

Epoch processing time (secs)    :  20.395309925079346
Total elapsed time (mins)       :  16.625536890824637
Estimated time remaining (mins) :  0.0

Saved model as *signs_set1*
In [25]:
# Train the model with the second training set and save out the session data

start_time = time.time()

with tf.Session() as sess:
    sess.run(tf.global_variables_initializer())
    num_examples = len(training_images_2)
    
    print('Training started, time to grab a cuppa...')
    print()
    print('Training for set #2: basic training images, with equalised image brightnesses')
    print('Set size      : ', num_examples)
    print('Epochs        : ', EPOCHS)
    print('Batch size    : ', BATCH_SIZE)
    print('Learning rate : ', rate)
    print()
    
    for i in range(EPOCHS):
        start_epoch = time.time()
        # take a shuffled copy of the training data for each epoch
        tr_images, tr_labels = shuffle(training_images_2, training_labels_2)
        for offset in range(i, num_examples, BATCH_SIZE):
            end = offset + BATCH_SIZE
            batch_x, batch_y = tr_images[offset:end], tr_labels[offset:end]
            sess.run(training_operation, feed_dict={X: batch_x, y: batch_y})
            
        validation_accuracy = evaluate(validation_images_2, validation_labels_2)
        print('EPOCH               :', i+1, ' of ', EPOCHS)
        print('Validation accuracy :', validation_accuracy)
        print()
        
        end_epoch = time.time()
        print('Epoch processing time (secs)    : ', end_epoch - start_epoch)
        print('Total elapsed time (mins)       : ', (end_epoch - start_time)/60.0)
        print('Estimated time remaining (mins) : ', (end_epoch - start_epoch) * (EPOCHS - (i+1)) / 60.0)
        print()
        
    # save the model
    try:
        saver
    except NameError:
        saver = tf.train.Saver()
    saver.save(sess, 'signs_set2')
    print('Saved model as *signs_set2*')
Training started, time to grab a cuppa...

Training for set #2: basic training images, with equalised image brightnesses
Set size      :  35288
Epochs        :  50
Batch size    :  128
Learning rate :  0.001

EPOCH               : 1  of  50
Validation accuracy : 0.814843152835

Epoch processing time (secs)    :  19.134544849395752
Total elapsed time (mins)       :  0.32257304986317953
Estimated time remaining (mins) :  15.626544960339864

EPOCH               : 2  of  50
Validation accuracy : 0.904361132805

Epoch processing time (secs)    :  17.96448278427124
Total elapsed time (mins)       :  0.6220055222511292
Estimated time remaining (mins) :  14.371586227416993

EPOCH               : 3  of  50
Validation accuracy : 0.946697270785

Epoch processing time (secs)    :  17.660322904586792
Total elapsed time (mins)       :  0.9163682500521342
Estimated time remaining (mins) :  13.833919608592987

EPOCH               : 4  of  50
Validation accuracy : 0.956388675887

Epoch processing time (secs)    :  19.35307288169861
Total elapsed time (mins)       :  1.238948945204417
Estimated time remaining (mins) :  14.837355875968933

EPOCH               : 5  of  50
Validation accuracy : 0.959704157255

Epoch processing time (secs)    :  17.92255425453186
Total elapsed time (mins)       :  1.5376783927281699
Estimated time remaining (mins) :  13.441915690898895

EPOCH               : 6  of  50
Validation accuracy : 0.970415710734

Epoch processing time (secs)    :  17.335234880447388
Total elapsed time (mins)       :  1.8266266584396362
Estimated time remaining (mins) :  12.712505578994751

EPOCH               : 7  of  50
Validation accuracy : 0.972200968669

Epoch processing time (secs)    :  17.344484090805054
Total elapsed time (mins)       :  2.1157283902168276
Estimated time remaining (mins) :  12.430213598410289

EPOCH               : 8  of  50
Validation accuracy : 0.976536598111

Epoch processing time (secs)    :  17.348846435546875
Total elapsed time (mins)       :  2.4049026052157085
Estimated time remaining (mins) :  12.144192504882813

EPOCH               : 9  of  50
Validation accuracy : 0.978831930934

Epoch processing time (secs)    :  17.92136788368225
Total elapsed time (mins)       :  2.7036137620608014
Estimated time remaining (mins) :  12.246268053849539

EPOCH               : 10  of  50
Validation accuracy : 0.978831930934

Epoch processing time (secs)    :  17.312793016433716
Total elapsed time (mins)       :  2.9921871860822042
Estimated time remaining (mins) :  11.54186201095581

EPOCH               : 11  of  50
Validation accuracy : 0.982402448355

Epoch processing time (secs)    :  17.203154802322388
Total elapsed time (mins)       :  3.2789309779802958
Estimated time remaining (mins) :  11.182050621509552

EPOCH               : 12  of  50
Validation accuracy : 0.985207855139

Epoch processing time (secs)    :  18.042362451553345
Total elapsed time (mins)       :  3.5796629389127097
Estimated time remaining (mins) :  11.426829552650451

EPOCH               : 13  of  50
Validation accuracy : 0.981892374394

Epoch processing time (secs)    :  17.27484655380249
Total elapsed time (mins)       :  3.8676023920377096
Estimated time remaining (mins) :  10.652822041511536

EPOCH               : 14  of  50
Validation accuracy : 0.977556746032

Epoch processing time (secs)    :  17.24223804473877
Total elapsed time (mins)       :  4.154998672008515
Estimated time remaining (mins) :  10.345342826843261

EPOCH               : 15  of  50
Validation accuracy : 0.979852078703

Epoch processing time (secs)    :  17.263493061065674
Total elapsed time (mins)       :  4.44275031487147
Estimated time remaining (mins) :  10.07037095228831

EPOCH               : 16  of  50
Validation accuracy : 0.978321856973

Epoch processing time (secs)    :  20.315627574920654
Total elapsed time (mins)       :  4.781368470191955
Estimated time remaining (mins) :  11.512188959121705

EPOCH               : 17  of  50
Validation accuracy : 0.981637337566

Epoch processing time (secs)    :  19.137221336364746
Total elapsed time (mins)       :  5.100323824087779
Estimated time remaining (mins) :  10.525471735000611

EPOCH               : 18  of  50
Validation accuracy : 0.984442744198

Epoch processing time (secs)    :  19.64510679244995
Total elapsed time (mins)       :  5.427769458293914
Estimated time remaining (mins) :  10.47739028930664

EPOCH               : 19  of  50
Validation accuracy : 0.986483040041

Epoch processing time (secs)    :  19.67665386199951
Total elapsed time (mins)       :  5.755733454227448
Estimated time remaining (mins) :  10.16627116203308

EPOCH               : 20  of  50
Validation accuracy : 0.976536597959

Epoch processing time (secs)    :  17.287037134170532
Total elapsed time (mins)       :  6.043877430756887
Estimated time remaining (mins) :  8.643518567085266

EPOCH               : 21  of  50
Validation accuracy : 0.978321856821

Epoch processing time (secs)    :  17.83644461631775
Total elapsed time (mins)       :  6.341179219881694
Estimated time remaining (mins) :  8.620948231220245

EPOCH               : 22  of  50
Validation accuracy : 0.955878603005

Epoch processing time (secs)    :  18.273998022079468
Total elapsed time (mins)       :  6.645765992005666
Estimated time remaining (mins) :  8.527865743637085

EPOCH               : 23  of  50
Validation accuracy : 0.986738077173

Epoch processing time (secs)    :  17.607287406921387
Total elapsed time (mins)       :  6.93924328883489
Estimated time remaining (mins) :  7.923279333114624

EPOCH               : 24  of  50
Validation accuracy : 0.978831930934

Epoch processing time (secs)    :  17.325103759765625
Total elapsed time (mins)       :  7.228026040395101
Estimated time remaining (mins) :  7.507544962565104

EPOCH               : 25  of  50
Validation accuracy : 0.980362152664

Epoch processing time (secs)    :  17.18897533416748
Total elapsed time (mins)       :  7.514534445603688
Estimated time remaining (mins) :  7.162073055903117

EPOCH               : 26  of  50
Validation accuracy : 0.986483040193

Epoch processing time (secs)    :  17.61310625076294
Total elapsed time (mins)       :  7.808113547166188
Estimated time remaining (mins) :  7.045242500305176

EPOCH               : 27  of  50
Validation accuracy : 0.988013262075

Epoch processing time (secs)    :  19.020642280578613
Total elapsed time (mins)       :  8.125149969259898
Estimated time remaining (mins) :  7.291246207555135

EPOCH               : 28  of  50
Validation accuracy : 0.982147411527

Epoch processing time (secs)    :  17.640208959579468
Total elapsed time (mins)       :  8.419180075327555
Estimated time remaining (mins) :  6.468076618512471

EPOCH               : 29  of  50
Validation accuracy : 0.986483040041

Epoch processing time (secs)    :  18.15494441986084
Total elapsed time (mins)       :  8.721787297725678
Estimated time remaining (mins) :  6.354230546951294

EPOCH               : 30  of  50
Validation accuracy : 0.98597296608

Epoch processing time (secs)    :  17.576546669006348
Total elapsed time (mins)       :  9.014750361442566
Estimated time remaining (mins) :  5.858848889668782

EPOCH               : 31  of  50
Validation accuracy : 0.988523335884

Epoch processing time (secs)    :  17.1309552192688
Total elapsed time (mins)       :  9.300289646784465
Estimated time remaining (mins) :  5.424802486101786

EPOCH               : 32  of  50
Validation accuracy : 0.989798520786

Epoch processing time (secs)    :  17.633668184280396
Total elapsed time (mins)       :  9.594211081663767
Estimated time remaining (mins) :  5.290100455284119

EPOCH               : 33  of  50
Validation accuracy : 0.985462892423

Epoch processing time (secs)    :  17.45528554916382
Total elapsed time (mins)       :  9.885158554712932
Estimated time remaining (mins) :  4.945664238929749

EPOCH               : 34  of  50
Validation accuracy : 0.987248151134

Epoch processing time (secs)    :  17.63479781150818
Total elapsed time (mins)       :  10.179100557168324
Estimated time remaining (mins) :  4.702612749735514

EPOCH               : 35  of  50
Validation accuracy : 0.989288446825

Epoch processing time (secs)    :  17.777810096740723
Total elapsed time (mins)       :  10.475425362586975
Estimated time remaining (mins) :  4.444452524185181

EPOCH               : 36  of  50
Validation accuracy : 0.987503187962

Epoch processing time (secs)    :  17.866896390914917
Total elapsed time (mins)       :  10.773232054710387
Estimated time remaining (mins) :  4.168942491213481

EPOCH               : 37  of  50
Validation accuracy : 0.980362152664

Epoch processing time (secs)    :  17.92612910270691
Total elapsed time (mins)       :  11.072029300530751
Estimated time remaining (mins) :  3.8839946389198303

EPOCH               : 38  of  50
Validation accuracy : 0.986738077173

Epoch processing time (secs)    :  17.223973035812378
Total elapsed time (mins)       :  11.359118783473969
Estimated time remaining (mins) :  3.4447946071624758

EPOCH               : 39  of  50
Validation accuracy : 0.979852078855

Epoch processing time (secs)    :  17.279361486434937
Total elapsed time (mins)       :  11.647136370340982
Estimated time remaining (mins) :  3.1678829391797385

EPOCH               : 40  of  50
Validation accuracy : 0.986228003364

Epoch processing time (secs)    :  17.50084137916565
Total elapsed time (mins)       :  11.938841764132182
Estimated time remaining (mins) :  2.9168068965276084

EPOCH               : 41  of  50
Validation accuracy : 0.984442744502

Epoch processing time (secs)    :  18.201523542404175
Total elapsed time (mins)       :  12.242226239045461
Estimated time remaining (mins) :  2.730228531360626

EPOCH               : 42  of  50
Validation accuracy : 0.983677633409

Epoch processing time (secs)    :  18.69295024871826
Total elapsed time (mins)       :  12.553800038496654
Estimated time remaining (mins) :  2.4923933664957683

EPOCH               : 43  of  50
Validation accuracy : 0.989798520786

Epoch processing time (secs)    :  17.599377870559692
Total elapsed time (mins)       :  12.847146463394164
Estimated time remaining (mins) :  2.0532607515652974

EPOCH               : 44  of  50
Validation accuracy : 0.978066819993

Epoch processing time (secs)    :  17.34756064414978
Total elapsed time (mins)       :  13.13630135456721
Estimated time remaining (mins) :  1.734756064414978

EPOCH               : 45  of  50
Validation accuracy : 0.987758224943

Epoch processing time (secs)    :  17.509764432907104
Total elapsed time (mins)       :  13.428156173229217
Estimated time remaining (mins) :  1.459147036075592

EPOCH               : 46  of  50
Validation accuracy : 0.984442744502

Epoch processing time (secs)    :  17.40108036994934
Total elapsed time (mins)       :  13.718199181556702
Estimated time remaining (mins) :  1.1600720246632894

EPOCH               : 47  of  50
Validation accuracy : 0.989798520786

Epoch processing time (secs)    :  17.50776743888855
Total elapsed time (mins)       :  14.010021018981934
Estimated time remaining (mins) :  0.8753883719444275

EPOCH               : 48  of  50
Validation accuracy : 0.985462892423

Epoch processing time (secs)    :  17.39978051185608
Total elapsed time (mins)       :  14.30004452864329
Estimated time remaining (mins) :  0.579992683728536

EPOCH               : 49  of  50
Validation accuracy : 0.985207855443

Epoch processing time (secs)    :  17.532384634017944
Total elapsed time (mins)       :  14.592278166611989
Estimated time remaining (mins) :  0.2922064105669657

EPOCH               : 50  of  50
Validation accuracy : 0.983167559296

Epoch processing time (secs)    :  17.4862117767334
Total elapsed time (mins)       :  14.883739805221557
Estimated time remaining (mins) :  0.0

Saved model as *signs_set2*
In [26]:
# Train the model with the third training set and save out the session data

start_time = time.time()

with tf.Session() as sess:
    sess.run(tf.global_variables_initializer())
    num_examples = len(training_images_3)
    
    print('Training started, time to grab a cuppa...')
    print()
    print('Training for set #3: basic training images, plus additional rotated images')
    print('Set size      : ', num_examples)
    print('Epochs        : ', EPOCHS)
    print('Batch size    : ', BATCH_SIZE)
    print('Learning rate : ', rate)
    print()
    
    for i in range(EPOCHS):
        start_epoch = time.time()
        # take a shuffled copy of the training data for each epoch
        tr_images, tr_labels = shuffle(training_images_3, training_labels_3)
        for offset in range(i, num_examples, BATCH_SIZE):
            end = offset + BATCH_SIZE
            batch_x, batch_y = tr_images[offset:end], tr_labels[offset:end]
            sess.run(training_operation, feed_dict={X: batch_x, y: batch_y})
            
        validation_accuracy = evaluate(validation_images_3, validation_labels_3)
        print('EPOCH               :', i+1, ' of ', EPOCHS)
        print('Validation accuracy :', validation_accuracy)
        print()
        
        end_epoch = time.time()
        print('Epoch processing time (secs)    : ', end_epoch - start_epoch)
        print('Total elapsed time (mins)       : ', (end_epoch - start_time)/60.0)
        print('Estimated time remaining (mins) : ', (end_epoch - start_epoch) * (EPOCHS - (i+1)) / 60.0)
        print()
        
    # save the model
    try:
        saver
    except NameError:
        saver = tf.train.Saver()
    saver.save(sess, 'signs_set3')
    print('Saved model as *signs_set3*')
Training started, time to grab a cuppa...

Training for set #3: basic training images, plus additional rotated images
Set size      :  43322
Epochs        :  50
Batch size    :  128
Learning rate :  0.001

EPOCH               : 1  of  50
Validation accuracy : 0.806681968703

Epoch processing time (secs)    :  23.56844735145569
Total elapsed time (mins)       :  0.39539825121561684
Estimated time remaining (mins) :  19.247565337022145

EPOCH               : 2  of  50
Validation accuracy : 0.877072174979

Epoch processing time (secs)    :  23.3859121799469
Total elapsed time (mins)       :  0.7851910630861918
Estimated time remaining (mins) :  18.70872974395752

EPOCH               : 3  of  50
Validation accuracy : 0.926549349488

Epoch processing time (secs)    :  24.315958499908447
Total elapsed time (mins)       :  1.1904791116714477
Estimated time remaining (mins) :  19.047500824928285

EPOCH               : 4  of  50
Validation accuracy : 0.927569497258

Epoch processing time (secs)    :  24.334792852401733
Total elapsed time (mins)       :  1.5960843483606975
Estimated time remaining (mins) :  18.65667452017466

EPOCH               : 5  of  50
Validation accuracy : 0.94873756678

Epoch processing time (secs)    :  23.759673357009888
Total elapsed time (mins)       :  1.9921023726463318
Estimated time remaining (mins) :  17.819755017757416

EPOCH               : 6  of  50
Validation accuracy : 0.951798010544

Epoch processing time (secs)    :  24.21056866645813
Total elapsed time (mins)       :  2.395638652642568
Estimated time remaining (mins) :  17.754417022069294

EPOCH               : 7  of  50
Validation accuracy : 0.955623564946

Epoch processing time (secs)    :  23.87700629234314
Total elapsed time (mins)       :  2.7936142484347024
Estimated time remaining (mins) :  17.111854509512582

EPOCH               : 8  of  50
Validation accuracy : 0.948737567403

Epoch processing time (secs)    :  23.032983779907227
Total elapsed time (mins)       :  3.1775307416915894
Estimated time remaining (mins) :  16.12308864593506

EPOCH               : 9  of  50
Validation accuracy : 0.96174445325

Epoch processing time (secs)    :  23.916006565093994
Total elapsed time (mins)       :  3.576157772541046
Estimated time remaining (mins) :  16.34260448614756

EPOCH               : 10  of  50
Validation accuracy : 0.967355266514

Epoch processing time (secs)    :  22.164861917495728
Total elapsed time (mins)       :  3.9456019202868142
Estimated time remaining (mins) :  14.776574611663818

EPOCH               : 11  of  50
Validation accuracy : 0.960979342613

Epoch processing time (secs)    :  24.008224964141846
Total elapsed time (mins)       :  4.345766484737396
Estimated time remaining (mins) :  15.6053462266922

EPOCH               : 12  of  50
Validation accuracy : 0.963784749245

Epoch processing time (secs)    :  21.982884883880615
Total elapsed time (mins)       :  4.712181146939596
Estimated time remaining (mins) :  13.922493759791056

EPOCH               : 13  of  50
Validation accuracy : 0.956133638906

Epoch processing time (secs)    :  23.281480312347412
Total elapsed time (mins)       :  5.100231790542603
Estimated time remaining (mins) :  14.356912859280904

EPOCH               : 14  of  50
Validation accuracy : 0.96735526697

Epoch processing time (secs)    :  24.009791374206543
Total elapsed time (mins)       :  5.500424734751383
Estimated time remaining (mins) :  14.405874824523925

EPOCH               : 15  of  50
Validation accuracy : 0.961234379289

Epoch processing time (secs)    :  24.905757904052734
Total elapsed time (mins)       :  5.915545880794525
Estimated time remaining (mins) :  14.528358777364096

EPOCH               : 16  of  50
Validation accuracy : 0.972711043405

Epoch processing time (secs)    :  23.825473070144653
Total elapsed time (mins)       :  6.312662982940674
Estimated time remaining (mins) :  13.501101406415303

EPOCH               : 17  of  50
Validation accuracy : 0.967100229989

Epoch processing time (secs)    :  22.70603060722351
Total elapsed time (mins)       :  6.691122575600942
Estimated time remaining (mins) :  12.488316833972931

EPOCH               : 18  of  50
Validation accuracy : 0.964549859258

Epoch processing time (secs)    :  24.346251010894775
Total elapsed time (mins)       :  7.0969190239906315
Estimated time remaining (mins) :  12.984667205810547

EPOCH               : 19  of  50
Validation accuracy : 0.971690895484

Epoch processing time (secs)    :  23.872499465942383
Total elapsed time (mins)       :  7.494817237059276
Estimated time remaining (mins) :  12.33412472407023

EPOCH               : 20  of  50
Validation accuracy : 0.964294823205

Epoch processing time (secs)    :  23.313965320587158
Total elapsed time (mins)       :  7.883409365018209
Estimated time remaining (mins) :  11.656982660293579

EPOCH               : 21  of  50
Validation accuracy : 0.977811782861

Epoch processing time (secs)    :  23.354034900665283
Total elapsed time (mins)       :  8.272671488920848
Estimated time remaining (mins) :  11.287783535321553

EPOCH               : 22  of  50
Validation accuracy : 0.964549860034

Epoch processing time (secs)    :  23.206982374191284
Total elapsed time (mins)       :  8.659479796886444
Estimated time remaining (mins) :  10.829925107955933

EPOCH               : 23  of  50
Validation accuracy : 0.980362152664

Epoch processing time (secs)    :  23.124572277069092
Total elapsed time (mins)       :  9.044916168848674
Estimated time remaining (mins) :  10.406057524681092

EPOCH               : 24  of  50
Validation accuracy : 0.970925784847

Epoch processing time (secs)    :  23.230371475219727
Total elapsed time (mins)       :  9.432114843527476
Estimated time remaining (mins) :  10.066494305928549

EPOCH               : 25  of  50
Validation accuracy : 0.972456006729

Epoch processing time (secs)    :  22.980631828308105
Total elapsed time (mins)       :  9.815153662363688
Estimated time remaining (mins) :  9.575263261795044

EPOCH               : 26  of  50
Validation accuracy : 0.975516450341

Epoch processing time (secs)    :  23.29225254058838
Total elapsed time (mins)       :  10.203386215368907
Estimated time remaining (mins) :  9.316901016235352

EPOCH               : 27  of  50
Validation accuracy : 0.976026524302

Epoch processing time (secs)    :  23.213850259780884
Total elapsed time (mins)       :  10.590308984120687
Estimated time remaining (mins) :  8.898642599582672

EPOCH               : 28  of  50
Validation accuracy : 0.970415710886

Epoch processing time (secs)    :  23.314566373825073
Total elapsed time (mins)       :  10.978908383846283
Estimated time remaining (mins) :  8.548674337069194

EPOCH               : 29  of  50
Validation accuracy : 0.958939046466

Epoch processing time (secs)    :  23.430575609207153
Total elapsed time (mins)       :  11.369441099961598
Estimated time remaining (mins) :  8.200701463222504

EPOCH               : 30  of  50
Validation accuracy : 0.976791635091

Epoch processing time (secs)    :  23.19178533554077
Total elapsed time (mins)       :  11.755997522672017
Estimated time remaining (mins) :  7.730595111846924

EPOCH               : 31  of  50
Validation accuracy : 0.977301709204

Epoch processing time (secs)    :  23.640044450759888
Total elapsed time (mins)       :  12.150025649865468
Estimated time remaining (mins) :  7.486014076073965

EPOCH               : 32  of  50
Validation accuracy : 0.973986228155

Epoch processing time (secs)    :  23.488970518112183
Total elapsed time (mins)       :  12.541537777582805
Estimated time remaining (mins) :  7.046691155433654

EPOCH               : 33  of  50
Validation accuracy : 0.979086967914

Epoch processing time (secs)    :  24.031882286071777
Total elapsed time (mins)       :  12.942093332608541
Estimated time remaining (mins) :  6.809033314387004

EPOCH               : 34  of  50
Validation accuracy : 0.976536597959

Epoch processing time (secs)    :  23.76443099975586
Total elapsed time (mins)       :  13.338192093372346
Estimated time remaining (mins) :  6.337181599934896

EPOCH               : 35  of  50
Validation accuracy : 0.958173935829

Epoch processing time (secs)    :  23.97683358192444
Total elapsed time (mins)       :  13.73783318599065
Estimated time remaining (mins) :  5.99420839548111

EPOCH               : 36  of  50
Validation accuracy : 0.977301708277

Epoch processing time (secs)    :  23.628753423690796
Total elapsed time (mins)       :  14.13167017698288
Estimated time remaining (mins) :  5.513375798861186

EPOCH               : 37  of  50
Validation accuracy : 0.978321856669

Epoch processing time (secs)    :  23.763524293899536
Total elapsed time (mins)       :  14.527758189042409
Estimated time remaining (mins) :  5.148763597011566

EPOCH               : 38  of  50
Validation accuracy : 0.979342004895

Epoch processing time (secs)    :  24.090493202209473
Total elapsed time (mins)       :  14.929290926456451
Estimated time remaining (mins) :  4.818098640441894

EPOCH               : 39  of  50
Validation accuracy : 0.969650599489

Epoch processing time (secs)    :  23.579524517059326
Total elapsed time (mins)       :  15.32230630715688
Estimated time remaining (mins) :  4.322912828127543

EPOCH               : 40  of  50
Validation accuracy : 0.972200969445

Epoch processing time (secs)    :  23.66068935394287
Total elapsed time (mins)       :  15.71667609612147
Estimated time remaining (mins) :  3.943448225657145

EPOCH               : 41  of  50
Validation accuracy : 0.974751339552

Epoch processing time (secs)    :  24.195889949798584
Total elapsed time (mins)       :  16.11996649503708
Estimated time remaining (mins) :  3.6293834924697874

EPOCH               : 42  of  50
Validation accuracy : 0.981382300586

Epoch processing time (secs)    :  23.6220965385437
Total elapsed time (mins)       :  16.513695764541627
Estimated time remaining (mins) :  3.149612871805827

EPOCH               : 43  of  50
Validation accuracy : 0.973221117366

Epoch processing time (secs)    :  24.769407272338867
Total elapsed time (mins)       :  16.926542977492016
Estimated time remaining (mins) :  2.889764181772868

EPOCH               : 44  of  50
Validation accuracy : 0.979342004743

Epoch processing time (secs)    :  24.47793459892273
Total elapsed time (mins)       :  17.334535257021585
Estimated time remaining (mins) :  2.447793459892273

EPOCH               : 45  of  50
Validation accuracy : 0.980872226625

Epoch processing time (secs)    :  24.073041677474976
Total elapsed time (mins)       :  17.73577816883723
Estimated time remaining (mins) :  2.006086806456248

EPOCH               : 46  of  50
Validation accuracy : 0.972711043253

Epoch processing time (secs)    :  27.841590881347656
Total elapsed time (mins)       :  18.19982884724935
Estimated time remaining (mins) :  1.8561060587565104

EPOCH               : 47  of  50
Validation accuracy : 0.980872226777

Epoch processing time (secs)    :  26.72706890106201
Total elapsed time (mins)       :  18.64530556599299
Estimated time remaining (mins) :  1.3363534450531005

EPOCH               : 48  of  50
Validation accuracy : 0.980362152512

Epoch processing time (secs)    :  24.179479598999023
Total elapsed time (mins)       :  19.048322455088297
Estimated time remaining (mins) :  0.8059826532999674

EPOCH               : 49  of  50
Validation accuracy : 0.972200969293

Epoch processing time (secs)    :  24.211990118026733
Total elapsed time (mins)       :  19.451883657773337
Estimated time remaining (mins) :  0.4035331686337789

EPOCH               : 50  of  50
Validation accuracy : 0.980107115684

Epoch processing time (secs)    :  24.26807451248169
Total elapsed time (mins)       :  19.856375924746196
Estimated time remaining (mins) :  0.0

Saved model as *signs_set3*
In [27]:
# Train the model with the fourth training set and save out the session data

start_time = time.time()

with tf.Session() as sess:
    sess.run(tf.global_variables_initializer())
    num_examples = len(training_images_4)
    
    print('Training started, time to grab a cuppa...')
    print()
    print('Training for set #4: basic training images, plus additional rotated images, with equalisation')
    print('Set size      : ', num_examples)
    print('Epochs        : ', EPOCHS)
    print('Batch size    : ', BATCH_SIZE)
    print('Learning rate : ', rate)
    print()
    
    for i in range(EPOCHS):
        start_epoch = time.time()
        # take a shuffled copy of the training data for each epoch
        tr_images, tr_labels = shuffle(training_images_4, training_labels_4)
        for offset in range(i, num_examples, BATCH_SIZE):
            end = offset + BATCH_SIZE
            batch_x, batch_y = tr_images[offset:end], tr_labels[offset:end]
            sess.run(training_operation, feed_dict={X: batch_x, y: batch_y})
            
        validation_accuracy = evaluate(validation_images_4, validation_labels_4)
        print('EPOCH               :', i+1, ' of ', EPOCHS)
        print('Validation accuracy :', validation_accuracy)
        print()
        
        end_epoch = time.time()
        print('Epoch processing time (secs)    : ', end_epoch - start_epoch)
        print('Total elapsed time (mins)       : ', (end_epoch - start_time)/60.0)
        print('Estimated time remaining (mins) : ', (end_epoch - start_epoch) * (EPOCHS - (i+1)) / 60.0)
        print()
        
    # save the model
    try:
        saver
    except NameError:
        saver = tf.train.Saver()
    saver.save(sess, 'signs_set4')
    print('Saved model as *signs_set4*')
Training started, time to grab a cuppa...

Training for set #4: basic training images, plus additional rotated images, with equalisation
Set size      :  43322
Epochs        :  50
Batch size    :  128
Learning rate :  0.001

EPOCH               : 1  of  50
Validation accuracy : 0.788319306573

Epoch processing time (secs)    :  22.5342218875885
Total elapsed time (mins)       :  0.37809836069742836
Estimated time remaining (mins) :  18.402947874863944

EPOCH               : 2  of  50
Validation accuracy : 0.908951797691

Epoch processing time (secs)    :  21.93052077293396
Total elapsed time (mins)       :  0.7436285535494487
Estimated time remaining (mins) :  17.544416618347167

EPOCH               : 3  of  50
Validation accuracy : 0.945422085883

Epoch processing time (secs)    :  23.624918460845947
Total elapsed time (mins)       :  1.1373985171318055
Estimated time remaining (mins) :  18.50618612766266

EPOCH               : 4  of  50
Validation accuracy : 0.946187196824

Epoch processing time (secs)    :  21.37234115600586
Total elapsed time (mins)       :  1.4936309496561686
Estimated time remaining (mins) :  16.385461552937826

EPOCH               : 5  of  50
Validation accuracy : 0.956643713946

Epoch processing time (secs)    :  20.975148677825928
Total elapsed time (mins)       :  1.8432453751564026
Estimated time remaining (mins) :  15.731361508369446

EPOCH               : 6  of  50
Validation accuracy : 0.960214231672

Epoch processing time (secs)    :  22.358819484710693
Total elapsed time (mins)       :  2.2159181237220764
Estimated time remaining (mins) :  16.396467622121175

EPOCH               : 7  of  50
Validation accuracy : 0.967865340778

Epoch processing time (secs)    :  20.948723077774048
Total elapsed time (mins)       :  2.56508713165919
Estimated time remaining (mins) :  15.013251539071401

EPOCH               : 8  of  50
Validation accuracy : 0.967865340474

Epoch processing time (secs)    :  20.996472358703613
Total elapsed time (mins)       :  2.915056073665619
Estimated time remaining (mins) :  14.69753065109253

EPOCH               : 9  of  50
Validation accuracy : 0.977811783013

Epoch processing time (secs)    :  21.622711181640625
Total elapsed time (mins)       :  3.27546493212382
Estimated time remaining (mins) :  14.775519307454427

EPOCH               : 10  of  50
Validation accuracy : 0.973731191023

Epoch processing time (secs)    :  21.442697048187256
Total elapsed time (mins)       :  3.632867236932119
Estimated time remaining (mins) :  14.29513136545817

EPOCH               : 11  of  50
Validation accuracy : 0.978321856973

Epoch processing time (secs)    :  21.171302795410156
Total elapsed time (mins)       :  3.985749582449595
Estimated time remaining (mins) :  13.761346817016602

EPOCH               : 12  of  50
Validation accuracy : 0.978831930782

Epoch processing time (secs)    :  20.893181085586548
Total elapsed time (mins)       :  4.333992874622345
Estimated time remaining (mins) :  13.23234802087148

EPOCH               : 13  of  50
Validation accuracy : 0.981892374394

Epoch processing time (secs)    :  21.459012746810913
Total elapsed time (mins)       :  4.691661393642425
Estimated time remaining (mins) :  13.233057860533396

EPOCH               : 14  of  50
Validation accuracy : 0.982657485335

Epoch processing time (secs)    :  23.604471445083618
Total elapsed time (mins)       :  5.085095870494842
Estimated time remaining (mins) :  14.16268286705017

EPOCH               : 15  of  50
Validation accuracy : 0.978831930782

Epoch processing time (secs)    :  21.186729192733765
Total elapsed time (mins)       :  5.438229918479919
Estimated time remaining (mins) :  12.35892536242803

EPOCH               : 16  of  50
Validation accuracy : 0.983677633257

Epoch processing time (secs)    :  20.903326511383057
Total elapsed time (mins)       :  5.786648197968801
Estimated time remaining (mins) :  11.845218356450399

EPOCH               : 17  of  50
Validation accuracy : 0.981892374546

Epoch processing time (secs)    :  20.99836039543152
Total elapsed time (mins)       :  6.136645905176798
Estimated time remaining (mins) :  11.549098217487336

EPOCH               : 18  of  50
Validation accuracy : 0.981892374546

Epoch processing time (secs)    :  21.39025568962097
Total elapsed time (mins)       :  6.493173730373383
Estimated time remaining (mins) :  11.408136367797852

EPOCH               : 19  of  50
Validation accuracy : 0.982912522468

Epoch processing time (secs)    :  22.580310583114624
Total elapsed time (mins)       :  6.8695426940917965
Estimated time remaining (mins) :  11.66649380127589

EPOCH               : 20  of  50
Validation accuracy : 0.979852078703

Epoch processing time (secs)    :  21.141313076019287
Total elapsed time (mins)       :  7.221923033396403
Estimated time remaining (mins) :  10.570656538009644

EPOCH               : 21  of  50
Validation accuracy : 0.980107115988

Epoch processing time (secs)    :  21.26615071296692
Total elapsed time (mins)       :  7.576386308670044
Estimated time remaining (mins) :  10.278639511267345

EPOCH               : 22  of  50
Validation accuracy : 0.983932670237

Epoch processing time (secs)    :  21.063174962997437
Total elapsed time (mins)       :  7.927465713024139
Estimated time remaining (mins) :  9.829481649398804

EPOCH               : 23  of  50
Validation accuracy : 0.98418770737

Epoch processing time (secs)    :  21.210657358169556
Total elapsed time (mins)       :  8.281005688508351
Estimated time remaining (mins) :  9.5447958111763

EPOCH               : 24  of  50
Validation accuracy : 0.982912522468

Epoch processing time (secs)    :  21.285300970077515
Total elapsed time (mins)       :  8.635786155859629
Estimated time remaining (mins) :  9.223630420366923

EPOCH               : 25  of  50
Validation accuracy : 0.983932670237

Epoch processing time (secs)    :  24.301915645599365
Total elapsed time (mins)       :  9.040845414002737
Estimated time remaining (mins) :  10.125798185666403

EPOCH               : 26  of  50
Validation accuracy : 0.984952818159

Epoch processing time (secs)    :  21.331472396850586
Total elapsed time (mins)       :  9.396397451559702
Estimated time remaining (mins) :  8.532588958740234

EPOCH               : 27  of  50
Validation accuracy : 0.988778372864

Epoch processing time (secs)    :  21.36961007118225
Total elapsed time (mins)       :  9.75258582830429
Estimated time remaining (mins) :  8.191683860619863

EPOCH               : 28  of  50
Validation accuracy : 0.985462892119

Epoch processing time (secs)    :  21.183576107025146
Total elapsed time (mins)       :  10.105663466453553
Estimated time remaining (mins) :  7.767311239242554

EPOCH               : 29  of  50
Validation accuracy : 0.98444274435

Epoch processing time (secs)    :  21.216017723083496
Total elapsed time (mins)       :  10.45928883155187
Estimated time remaining (mins) :  7.425606203079224

EPOCH               : 30  of  50
Validation accuracy : 0.975261413057

Epoch processing time (secs)    :  21.264874935150146
Total elapsed time (mins)       :  10.813728706041973
Estimated time remaining (mins) :  7.088291645050049

EPOCH               : 31  of  50
Validation accuracy : 0.983932670237

Epoch processing time (secs)    :  21.463504552841187
Total elapsed time (mins)       :  11.171478390693665
Estimated time remaining (mins) :  6.7967764417330425

EPOCH               : 32  of  50
Validation accuracy : 0.986738077021

Epoch processing time (secs)    :  21.15084171295166
Total elapsed time (mins)       :  11.524017028013866
Estimated time remaining (mins) :  6.345252513885498

EPOCH               : 33  of  50
Validation accuracy : 0.984952818311

Epoch processing time (secs)    :  21.477192401885986
Total elapsed time (mins)       :  11.881996774673462
Estimated time remaining (mins) :  6.085204513867696

EPOCH               : 34  of  50
Validation accuracy : 0.981127263453

Epoch processing time (secs)    :  21.63392734527588
Total elapsed time (mins)       :  12.24258974790573
Estimated time remaining (mins) :  5.769047292073568

EPOCH               : 35  of  50
Validation accuracy : 0.989543483805

Epoch processing time (secs)    :  21.139288902282715
Total elapsed time (mins)       :  12.594936339060466
Estimated time remaining (mins) :  5.284822225570679

EPOCH               : 36  of  50
Validation accuracy : 0.988268298903

Epoch processing time (secs)    :  21.50881028175354
Total elapsed time (mins)       :  12.953442307313283
Estimated time remaining (mins) :  5.018722399075826

EPOCH               : 37  of  50
Validation accuracy : 0.987248150982

Epoch processing time (secs)    :  21.720184564590454
Total elapsed time (mins)       :  13.315471867720285
Estimated time remaining (mins) :  4.706039988994599

EPOCH               : 38  of  50
Validation accuracy : 0.990053557766

Epoch processing time (secs)    :  22.007998943328857
Total elapsed time (mins)       :  13.68229558467865
Estimated time remaining (mins) :  4.401599788665772

EPOCH               : 39  of  50
Validation accuracy : 0.987248150982

Epoch processing time (secs)    :  22.550654649734497
Total elapsed time (mins)       :  14.058167111873626
Estimated time remaining (mins) :  4.134286685784658

EPOCH               : 40  of  50
Validation accuracy : 0.984952818463

Epoch processing time (secs)    :  22.80929732322693
Total elapsed time (mins)       :  14.438349354267121
Estimated time remaining (mins) :  3.801549553871155

EPOCH               : 41  of  50
Validation accuracy : 0.983167559296

Epoch processing time (secs)    :  20.926862716674805
Total elapsed time (mins)       :  14.78715744415919
Estimated time remaining (mins) :  3.1390294075012206

EPOCH               : 42  of  50
Validation accuracy : 0.988013261923

Epoch processing time (secs)    :  21.720099449157715
Total elapsed time (mins)       :  15.149181997776031
Estimated time remaining (mins) :  2.8960132598876953

EPOCH               : 43  of  50
Validation accuracy : 0.987758224943

Epoch processing time (secs)    :  21.475345134735107
Total elapsed time (mins)       :  15.507130841414133
Estimated time remaining (mins) :  2.5054569323857625

EPOCH               : 44  of  50
Validation accuracy : 0.988778372864

Epoch processing time (secs)    :  21.651626348495483
Total elapsed time (mins)       :  15.86801985502243
Estimated time remaining (mins) :  2.1651626348495485

EPOCH               : 45  of  50
Validation accuracy : 0.979597041723

Epoch processing time (secs)    :  21.771220207214355
Total elapsed time (mins)       :  16.230896604061126
Estimated time remaining (mins) :  1.8142683506011963

EPOCH               : 46  of  50
Validation accuracy : 0.987248150982

Epoch processing time (secs)    :  21.590153217315674
Total elapsed time (mins)       :  16.590758923689524
Estimated time remaining (mins) :  1.439343547821045

EPOCH               : 47  of  50
Validation accuracy : 0.990818668859

Epoch processing time (secs)    :  21.487080812454224
Total elapsed time (mins)       :  16.948901204268136
Estimated time remaining (mins) :  1.0743540406227112

EPOCH               : 48  of  50
Validation accuracy : 0.991583779648

Epoch processing time (secs)    :  22.435100078582764
Total elapsed time (mins)       :  17.322847859064737
Estimated time remaining (mins) :  0.7478366692860922

EPOCH               : 49  of  50
Validation accuracy : 0.987758225095

Epoch processing time (secs)    :  22.038561582565308
Total elapsed time (mins)       :  17.690182260672252
Estimated time remaining (mins) :  0.3673093597094218

EPOCH               : 50  of  50
Validation accuracy : 0.982147411375

Epoch processing time (secs)    :  21.781235456466675
Total elapsed time (mins)       :  18.053229014078777
Estimated time remaining (mins) :  0.0

Saved model as *signs_set4*

Question 4

How did you train your model? (Type of optimizer, batch size, epochs, hyperparameters, etc.)

Answer:

The choice of optimiser was taken from the example LeNet model provision, which uses the AdamOptimiser

Adam optimiser: see https://arxiv.org/abs/1412.6980

Note that the AdamOptimiser is quite a recent development, but considered to be computationally efficient and not requiring much in the way of tuning of input parameters (a good thing!).

Batch size and epochs were determined by trial-and-error, starting from settings found to be reasonable in previous projects. Similarly the learning rate was defaulted to 0.001, which is a typical starting point for the Adam optimiser.

Note that the number of epochs used in the 4 runs of model training above is a compromise between accuracy and computing time, as well as potential for over-fitting. Observation of the validation accuracy results over the 50 epochs shows that some models may be over-fitting (some drop-off in validation accuracy), whilst some may not yet be reaching the best level of performance. However, the network does not contain any tactics to address over-fitting and therefore large epochs settings are not advisable (may not yield better results, risk over-fit).

In [28]:
# Check the model on the test set as a final appraisal of accuracy

# Have X_test, y_test as final test data

# Can load in session data and run the evaluate function on the test data

# Get back the saved state for model #1
saver = tf.train.Saver()
# Launch the graph
with tf.Session() as sess:
    saver.restore(sess, '/home/neil/Documents/CarND-Traffic-Sign-Classifier-Project/signs_set1')

    test_accuracy = sess.run(accuracy_operation, feed_dict={X: X_test, y: y_test})
print('Final Test, Set #1; basic data')
print('Test Accuracy: {}'.format(test_accuracy))

# Set #2
saver = tf.train.Saver()
# Launch the graph
with tf.Session() as sess:
    saver.restore(sess, '/home/neil/Documents/CarND-Traffic-Sign-Classifier-Project/signs_set2')

    test_accuracy = sess.run(accuracy_operation, feed_dict={X: X_test, y: y_test})
print('Final Test, Set #2; basic data, equalised images')
print('Test Accuracy: {}'.format(test_accuracy))

# Need also to equalise the test image set as a pre-processing operation
test_eq = []
for idx in range(len(X_test)):
    test_eq.append(equalise_image(X_test[idx]))
saver = tf.train.Saver()
with tf.Session() as sess:
    saver.restore(sess, '/home/neil/Documents/CarND-Traffic-Sign-Classifier-Project/signs_set2')
    test_accuracy = sess.run(accuracy_operation, feed_dict={X: test_eq, y: y_test})
print('Final Test, Set #2; basic data, equalised images, equalised *test* images')
print('Test Accuracy: {}'.format(test_accuracy))
    

# Set #3
saver = tf.train.Saver()
# Launch the graph
with tf.Session() as sess:
    saver.restore(sess, '/home/neil/Documents/CarND-Traffic-Sign-Classifier-Project/signs_set3')

    test_accuracy = sess.run(accuracy_operation, feed_dict={X: X_test, y: y_test})
print('Final Test, Set #3; basic data, plus additional rotated images')
print('Test Accuracy: {}'.format(test_accuracy))

#Set #4
saver = tf.train.Saver()
# Launch the graph
with tf.Session() as sess:
    saver.restore(sess, '/home/neil/Documents/CarND-Traffic-Sign-Classifier-Project/signs_set4')

    test_accuracy = sess.run(accuracy_operation, feed_dict={X: X_test, y: y_test})
print('Final Test, Set #4; basic data, plus extra images, plus equalisation')
print('Test Accuracy: {}'.format(test_accuracy))
saver = tf.train.Saver()
with tf.Session() as sess:
    saver.restore(sess, '/home/neil/Documents/CarND-Traffic-Sign-Classifier-Project/signs_set4')
    test_accuracy = sess.run(accuracy_operation, feed_dict={X: test_eq, y: y_test})
print('Final Test, Set #4; basic data, equalised images, equalised *test* images')
print('Test Accuracy: {}'.format(test_accuracy))
Final Test, Set #1; basic data
Test Accuracy: 0.8964370489120483
Final Test, Set #2; basic data, equalised images
Test Accuracy: 0.9038796424865723
/home/neil/anaconda3/envs/CarND-Traffic-Sign-Classifier-Project/lib/python3.5/site-packages/skimage/exposure/exposure.py:63: UserWarning: This might be a color image. The histogram will be computed on the flattened image. You can instead apply this function to each color channel.
  warn("This might be a color image. The histogram will be "
Final Test, Set #2; basic data, equalised images, equalised *test* images
Test Accuracy: 0.9337292313575745
Final Test, Set #3; basic data, plus additional rotated images
Test Accuracy: 0.9178938865661621
Final Test, Set #4; basic data, plus extra images, plus equalisation
Test Accuracy: 0.8830562233924866
Final Test, Set #4; basic data, equalised images, equalised *test* images
Test Accuracy: 0.91464763879776

Reflections on the model test results

The models are all showing some evidence of over-fitting, as per the values below (i.e. test set accuracy lower than the validation set accuracy).

Set #1: final validation accuracy 96.8%, test set accuracy 89.6%

Set #2: final validation accuracy 98.3%, test set accuracy 93.4%

Set #3: final validation accuracy 98.0%, test set accuracy 91.8%

Set #4: final validation accuracy 98.2%, test set accuracy 91.5%

This suggests that some tactics to reduce over-fit may improve the test (real) performance of the network.

The ADAM optimiser doesn't appear to allow L2 regularization within the optimisation, so this option isn't available (would require a different optimiser to be chosen, such as ProximalGradientDescentOptimizer, which would allow L1/L2 params - but this would require more trial-and-error work for the L2 params). Instead, drop-out can be used on the convolutional layers to randomly drop a proportion of the inputs at each stage. This will avoid the model becoming overly dependent on any particular training images.

Of the 4 sets of training data used (i.e. of the 4 options for pre-processing), set #2 appears to be giving the best results (brightness equalisation, but without additional images added).

In [61]:
# Add drop-out after the convolutional stages in the LeNet model
# Add L2 regularisation after the first fully connected layer

tf.reset_default_graph()

# Hyper-parameters
EPOCHS = 100 #150        # Higher than standard initial setting, sacrificing the length of time to train for accuracy
BATCH_SIZE = 128    # Larger batches mean faster training, but may be limited by memory
mu = 0              # mean value of distribution used for random assignment of initial weights
sigma = 0.1         # standard deviation used for random assignment of initial weights - want this to be small
rate = 0.001        # learning rate, 0.001 is a good initial setting for a LeNet-type network
keep = 0.5

keep_prob = tf.placeholder(tf.float32) # probability to keep units

# Set up the network operations
# The input X is going to be the batch of training images
def SignIdentificationNetwork2(X):
    
    # First convolutional layer - output 28x28x6 (further reduced by pooling step)
    conv1_W = tf.Variable(tf.truncated_normal(shape=(5,5,3,6),mean=mu,stddev=sigma))
    conv1_b = tf.Variable(tf.zeros(6))
    conv1   = tf.nn.conv2d(X, conv1_W, strides=[1,1,1,1], padding='VALID') + conv1_b
    # Activation - relu function
    conv1   = tf.nn.relu(conv1) 
    # Drop-out
    conv1   = tf.nn.dropout(conv1, keep_prob)
    # Pooling step - get down to 14x14x6 output, using max_pool with 2x2 kernels
    conv1   = tf.nn.max_pool(conv1, ksize=[1,2,2,1], strides=[1,2,2,1], padding='VALID')  
    
    # Second convolutional layer - output 10x10x16 (further reduced by pooling step)
    conv2_W = tf.Variable(tf.truncated_normal(shape=(5,5,6,16), mean=mu, stddev=sigma))
    conv2_b = tf.Variable(tf.zeros(16))
    conv2   = tf.nn.conv2d(conv1, conv2_W, strides=[1,1,1,1], padding='VALID') + conv2_b
    # Activation - relu again
    conv2   = tf.nn.relu(conv2)
    # Drop-out
    conv2   = tf.nn.dropout(conv2, keep_prob)
    # Pooling step - get down to 5x5x16 output
    conv2   = tf.nn.max_pool(conv2, ksize=[1,2,2,1], strides=[1,2,2,1], padding='VALID')
    
    # Flatten step - go from 5x5x16 input to 1x400 output
    flat1   = flatten(conv2)
    
    # First fully connected layer - input 400, output 120
    full1_W = tf.Variable(tf.truncated_normal(shape=(400,120), mean=mu, stddev=sigma))
    full1_b = tf.Variable(tf.zeros(120))
    full1   = tf.matmul(flat1, full1_W) + full1_b
    # Activation - relu again
    full1   = tf.nn.relu(full1)
    
    # Second fully connected layer - input 120, output 84
    full2_W = tf.Variable(tf.truncated_normal(shape=(120,84), mean=mu, stddev=sigma))
    full2_b = tf.Variable(tf.zeros(84))
    full2   = tf.matmul(full1, full2_W) + full2_b
    # Activation - relu again
    full2   = tf.nn.relu(full2)
    
    # Third fully connected layer - input 84, output 43 (number of classes required)
    full3_W = tf.Variable(tf.truncated_normal(shape=(84,43), mean=mu, stddev=sigma))
    full3_b = tf.Variable(tf.zeros(43))
    logits  = tf.matmul(full2, full3_W) + full3_b
    
    return logits

# Set up placeholders
X = tf.placeholder(tf.float32, (None,32,32,3))      # dimensions set for 32x32x3 images
y = tf.placeholder(tf.int32, (None))
one_hot_y = tf.one_hot(y, 43)                       # sized for 43 output classes

# Set up the training pipeline and optimiser
logits = SignIdentificationNetwork2(X)
cross_entropy = tf.nn.softmax_cross_entropy_with_logits(logits, one_hot_y)
loss_operation = tf.reduce_mean(cross_entropy)
optimizer = tf.train.AdamOptimizer(learning_rate = rate)      # using the Adam optimiser - built in, better than SGD
training_operation = optimizer.minimize(loss_operation)

# Set up the evaluation function and related operations

# correct prediction if prediction matches the one-hot value
correct_prediction = tf.equal(tf.argmax(logits, 1), tf.argmax(one_hot_y, 1))
accuracy_operation = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))

# standard evaluation function
def evaluate(X_data, y_data):
    num_examples = len(X_data)
    total_accuracy = 0
    sess = tf.get_default_session()
    for offset in range(0, num_examples, BATCH_SIZE):
        batch_x, batch_y = X_data[offset:offset+BATCH_SIZE], y_data[offset:offset+BATCH_SIZE]
        accuracy = sess.run(accuracy_operation, feed_dict={X: batch_x, y: batch_y, keep_prob: 1.0})
                                                                                    # keep all values when evaluating
        total_accuracy += (accuracy * len(batch_x))
    return total_accuracy / num_examples

# Train the model with the second training set and save out the session data

start_time = time.time()

with tf.Session() as sess:
    sess.run(tf.global_variables_initializer())
    num_examples = len(training_images_2)
    
    print('Training started, time to grab a cuppa...')
    print()
    print('Training for set #2: basic training images, with equalised image brightnesses')
    print('Using drop-out in model training')
    print('Set size      : ', num_examples)
    print('Epochs        : ', EPOCHS)
    print('Batch size    : ', BATCH_SIZE)
    print('Learning rate : ', rate)
    print('Drop-out rate : ', 1.0-keep)
    print()
    
    for i in range(EPOCHS):
        start_epoch = time.time()
        # take a shuffled copy of the training data for each epoch
        tr_images, tr_labels = shuffle(training_images_2, training_labels_2)
        for offset in range(i, num_examples, BATCH_SIZE):
            end = offset + BATCH_SIZE
            batch_x, batch_y = tr_images[offset:end], tr_labels[offset:end]
            sess.run(training_operation, feed_dict={X: batch_x, y: batch_y, keep_prob: keep})
                                                                                # throw away with 50% probability
        validation_accuracy = evaluate(validation_images_2, validation_labels_2)
        print('EPOCH               :', i+1, ' of ', EPOCHS)
        print('Validation accuracy :', validation_accuracy)
        print()
        
        end_epoch = time.time()
        print('Epoch processing time (secs)    : ', end_epoch - start_epoch)
        print('Total elapsed time (mins)       : ', (end_epoch - start_time)/60.0)
        print('Estimated time remaining (mins) : ', (end_epoch - start_epoch) * (EPOCHS - (i+1)) / 60.0)
        print()
        
    # save the model
    saver = tf.train.Saver()
    saver.save(sess, 'signs_dropout_150')
    print('Saved model as *signs_dropout_150*')
    
    # check the accuracy on the test data set
    test_accuracy = sess.run(accuracy_operation, feed_dict={X: test_eq, y: y_test, keep_prob: 1.0})
    print('Final Test, Set #2; basic data, equalised images, drop-out in pipeline, equalised *test* images')
    print('Using drop-out model')
    print('Test Accuracy: {}'.format(test_accuracy))
Training started, time to grab a cuppa...

Training for set #2: basic training images, with equalised image brightnesses
Using drop-out in model training
Set size      :  35288
Epochs        :  100
Batch size    :  128
Learning rate :  0.001
Drop-out rate :  0.5

EPOCH               : 1  of  100
Validation accuracy : 0.811272634958

Epoch processing time (secs)    :  21.532721996307373
Total elapsed time (mins)       :  0.36131323178609215
Estimated time remaining (mins) :  35.52899129390717

EPOCH               : 2  of  100
Validation accuracy : 0.902575873015

Epoch processing time (secs)    :  20.74010467529297
Total elapsed time (mins)       :  0.7070069948832194
Estimated time remaining (mins) :  33.87550430297851

EPOCH               : 3  of  100
Validation accuracy : 0.939556235639

Epoch processing time (secs)    :  20.178058624267578
Total elapsed time (mins)       :  1.0433344721794129
Estimated time remaining (mins) :  32.62119477589925

EPOCH               : 4  of  100
Validation accuracy : 0.952308084353

Epoch processing time (secs)    :  24.560000896453857
Total elapsed time (mins)       :  1.4526930133501688
Estimated time remaining (mins) :  39.29600143432617

EPOCH               : 5  of  100
Validation accuracy : 0.950012752457

Epoch processing time (secs)    :  21.93332052230835
Total elapsed time (mins)       :  1.8182507753372192
Estimated time remaining (mins) :  34.727757493654885

EPOCH               : 6  of  100
Validation accuracy : 0.964039786225

Epoch processing time (secs)    :  20.038216829299927
Total elapsed time (mins)       :  2.152246419588725
Estimated time remaining (mins) :  31.393206365903218

EPOCH               : 7  of  100
Validation accuracy : 0.966080082068

Epoch processing time (secs)    :  19.289313554763794
Total elapsed time (mins)       :  2.4737640301386516
Estimated time remaining (mins) :  29.89843600988388

EPOCH               : 8  of  100
Validation accuracy : 0.979086968066

Epoch processing time (secs)    :  19.085065841674805
Total elapsed time (mins)       :  2.791874849796295
Estimated time remaining (mins) :  29.263767623901366

EPOCH               : 9  of  100
Validation accuracy : 0.976026524302

Epoch processing time (secs)    :  19.172397136688232
Total elapsed time (mins)       :  3.1114384651184084
Estimated time remaining (mins) :  29.078135657310487

EPOCH               : 10  of  100
Validation accuracy : 0.9688854887

Epoch processing time (secs)    :  19.22177505493164
Total elapsed time (mins)       :  3.4318234443664553
Estimated time remaining (mins) :  28.83266258239746

EPOCH               : 11  of  100
Validation accuracy : 0.969905636773

Epoch processing time (secs)    :  19.046977758407593
Total elapsed time (mins)       :  3.7493019819259645
Estimated time remaining (mins) :  28.253017008304596

EPOCH               : 12  of  100
Validation accuracy : 0.98444274435

Epoch processing time (secs)    :  19.255248069763184
Total elapsed time (mins)       :  4.0702579220136
Estimated time remaining (mins) :  28.241030502319337

EPOCH               : 13  of  100
Validation accuracy : 0.980362152816

Epoch processing time (secs)    :  19.2195041179657
Total elapsed time (mins)       :  4.390608624617259
Estimated time remaining (mins) :  27.86828097105026

EPOCH               : 14  of  100
Validation accuracy : 0.983932670541

Epoch processing time (secs)    :  19.149598121643066
Total elapsed time (mins)       :  4.70979304711024
Estimated time remaining (mins) :  27.447757307688395

EPOCH               : 15  of  100
Validation accuracy : 0.981892374698

Epoch processing time (secs)    :  19.02168869972229
Total elapsed time (mins)       :  5.0268471797307335
Estimated time remaining (mins) :  26.94739232460658

EPOCH               : 16  of  100
Validation accuracy : 0.977811783013

Epoch processing time (secs)    :  19.11246418952942
Total elapsed time (mins)       :  5.345409258206685
Estimated time remaining (mins) :  26.757449865341187

EPOCH               : 17  of  100
Validation accuracy : 0.984442744502

Epoch processing time (secs)    :  19.12262749671936
Total elapsed time (mins)       :  5.664145727952321
Estimated time remaining (mins) :  26.45296803712845

EPOCH               : 18  of  100
Validation accuracy : 0.985462892271

Epoch processing time (secs)    :  19.31966233253479
Total elapsed time (mins)       :  5.986166087786357
Estimated time remaining (mins) :  26.40353852113088

EPOCH               : 19  of  100
Validation accuracy : 0.98444274435

Epoch processing time (secs)    :  20.76035237312317
Total elapsed time (mins)       :  6.332198715209961
Estimated time remaining (mins) :  28.02647570371628

EPOCH               : 20  of  100
Validation accuracy : 0.987503188114

Epoch processing time (secs)    :  19.102607250213623
Total elapsed time (mins)       :  6.650599765777588
Estimated time remaining (mins) :  25.470143000284832

EPOCH               : 21  of  100
Validation accuracy : 0.987503188266

Epoch processing time (secs)    :  19.220006465911865
Total elapsed time (mins)       :  6.970960660775503
Estimated time remaining (mins) :  25.306341846783955

EPOCH               : 22  of  100
Validation accuracy : 0.988778373016

Epoch processing time (secs)    :  19.18327283859253
Total elapsed time (mins)       :  7.290707544485728
Estimated time remaining (mins) :  24.93825469017029

EPOCH               : 23  of  100
Validation accuracy : 0.987248151134

Epoch processing time (secs)    :  19.14351511001587
Total elapsed time (mins)       :  7.6097917556762695
Estimated time remaining (mins) :  24.5675110578537

EPOCH               : 24  of  100
Validation accuracy : 0.990563631879

Epoch processing time (secs)    :  19.026856184005737
Total elapsed time (mins)       :  7.926931015650431
Estimated time remaining (mins) :  24.100684499740602

EPOCH               : 25  of  100
Validation accuracy : 0.990563631879

Epoch processing time (secs)    :  19.21284008026123
Total elapsed time (mins)       :  8.247169919808705
Estimated time remaining (mins) :  24.016050100326538

EPOCH               : 26  of  100
Validation accuracy : 0.988778373016

Epoch processing time (secs)    :  20.1165452003479
Total elapsed time (mins)       :  8.582470834255219
Estimated time remaining (mins) :  24.810405747095743

EPOCH               : 27  of  100
Validation accuracy : 0.988268299055

Epoch processing time (secs)    :  19.093270301818848
Total elapsed time (mins)       :  8.900715823968252
Estimated time remaining (mins) :  23.230145533879597

EPOCH               : 28  of  100
Validation accuracy : 0.989033409996

Epoch processing time (secs)    :  19.105322122573853
Total elapsed time (mins)       :  9.219164562225341
Estimated time remaining (mins) :  22.926386547088622

EPOCH               : 29  of  100
Validation accuracy : 0.99311400153

Epoch processing time (secs)    :  19.435654640197754
Total elapsed time (mins)       :  9.543117880821228
Estimated time remaining (mins) :  22.998857990900675

EPOCH               : 30  of  100
Validation accuracy : 0.986738077325

Epoch processing time (secs)    :  19.037044525146484
Total elapsed time (mins)       :  9.8604265888532
Estimated time remaining (mins) :  22.209885279337566

EPOCH               : 31  of  100
Validation accuracy : 0.990563631879

Epoch processing time (secs)    :  19.130866050720215
Total elapsed time (mins)       :  10.179303507010141
Estimated time remaining (mins) :  22.000495958328248

EPOCH               : 32  of  100
Validation accuracy : 0.992348890741

Epoch processing time (secs)    :  19.375762701034546
Total elapsed time (mins)       :  10.502255686124165
Estimated time remaining (mins) :  21.95919772783915

EPOCH               : 33  of  100
Validation accuracy : 0.990563631879

Epoch processing time (secs)    :  19.13996982574463
Total elapsed time (mins)       :  10.821280479431152
Estimated time remaining (mins) :  21.372966305414835

EPOCH               : 34  of  100
Validation accuracy : 0.9915837798

Epoch processing time (secs)    :  19.04819393157959
Total elapsed time (mins)       :  11.138775980472564
Estimated time remaining (mins) :  20.953013324737547

EPOCH               : 35  of  100
Validation accuracy : 0.988523336036

Epoch processing time (secs)    :  19.135378122329712
Total elapsed time (mins)       :  11.45772529443105
Estimated time remaining (mins) :  20.72999296585719

EPOCH               : 36  of  100
Validation accuracy : 0.990308594898

Epoch processing time (secs)    :  19.9088077545166
Total elapsed time (mins)       :  11.78956386645635
Estimated time remaining (mins) :  21.23606160481771

EPOCH               : 37  of  100
Validation accuracy : 0.988268299055

Epoch processing time (secs)    :  18.985451698303223
Total elapsed time (mins)       :  12.106012777487438
Estimated time remaining (mins) :  19.934724283218383

EPOCH               : 38  of  100
Validation accuracy : 0.989543484109

Epoch processing time (secs)    :  19.325302124023438
Total elapsed time (mins)       :  12.42812873919805
Estimated time remaining (mins) :  19.969478861490884

EPOCH               : 39  of  100
Validation accuracy : 0.989288446977

Epoch processing time (secs)    :  19.6190083026886
Total elapsed time (mins)       :  12.75513885418574
Estimated time remaining (mins) :  19.945991774400074

EPOCH               : 40  of  100
Validation accuracy : 0.990563631879

Epoch processing time (secs)    :  19.153441429138184
Total elapsed time (mins)       :  13.07438854376475
Estimated time remaining (mins) :  19.153441429138184

EPOCH               : 41  of  100
Validation accuracy : 0.99183881678

Epoch processing time (secs)    :  19.096442699432373
Total elapsed time (mins)       :  13.392689367135366
Estimated time remaining (mins) :  18.778168654441835

EPOCH               : 42  of  100
Validation accuracy : 0.99183881678

Epoch processing time (secs)    :  19.21866273880005
Total elapsed time (mins)       :  13.713023765881855
Estimated time remaining (mins) :  18.578040647506715

EPOCH               : 43  of  100
Validation accuracy : 0.992858964702

Epoch processing time (secs)    :  19.648832321166992
Total elapsed time (mins)       :  14.04052627881368
Estimated time remaining (mins) :  18.666390705108643

EPOCH               : 44  of  100
Validation accuracy : 0.99132874282

Epoch processing time (secs)    :  19.260908365249634
Total elapsed time (mins)       :  14.36157293319702
Estimated time remaining (mins) :  17.976847807566326

EPOCH               : 45  of  100
Validation accuracy : 0.99183881678

Epoch processing time (secs)    :  19.46778154373169
Total elapsed time (mins)       :  14.686057372887928
Estimated time remaining (mins) :  17.845466415087383

EPOCH               : 46  of  100
Validation accuracy : 0.99311400153

Epoch processing time (secs)    :  19.405962705612183
Total elapsed time (mins)       :  15.009512754281362
Estimated time remaining (mins) :  17.465366435050964

EPOCH               : 47  of  100
Validation accuracy : 0.993879112471

Epoch processing time (secs)    :  19.108368396759033
Total elapsed time (mins)       :  15.32800874710083
Estimated time remaining (mins) :  16.87905875047048

EPOCH               : 48  of  100
Validation accuracy : 0.993369038663

Epoch processing time (secs)    :  19.19292449951172
Total elapsed time (mins)       :  15.647911930084229
Estimated time remaining (mins) :  16.633867899576824

EPOCH               : 49  of  100
Validation accuracy : 0.993624075643

Epoch processing time (secs)    :  19.55098032951355
Total elapsed time (mins)       :  15.973785614967346
Estimated time remaining (mins) :  16.618333280086517

EPOCH               : 50  of  100
Validation accuracy : 0.9915837798

Epoch processing time (secs)    :  19.285577535629272
Total elapsed time (mins)       :  16.295239702860513
Estimated time remaining (mins) :  16.071314613024395

EPOCH               : 51  of  100
Validation accuracy : 0.993624075491

Epoch processing time (secs)    :  19.29740571975708
Total elapsed time (mins)       :  16.616890716552735
Estimated time remaining (mins) :  15.759548004468282

EPOCH               : 52  of  100
Validation accuracy : 0.995154297373

Epoch processing time (secs)    :  19.09886336326599
Total elapsed time (mins)       :  16.935230429967245
Estimated time remaining (mins) :  15.279090690612794

EPOCH               : 53  of  100
Validation accuracy : 0.986993114306

Epoch processing time (secs)    :  19.4660325050354
Total elapsed time (mins)       :  17.259690233071645
Estimated time remaining (mins) :  15.248392128944397

EPOCH               : 54  of  100
Validation accuracy : 0.990308594746

Epoch processing time (secs)    :  19.30184531211853
Total elapsed time (mins)       :  17.58141666650772
Estimated time remaining (mins) :  14.79808140595754

EPOCH               : 55  of  100
Validation accuracy : 0.990818668859

Epoch processing time (secs)    :  19.407370805740356
Total elapsed time (mins)       :  17.904886603355408
Estimated time remaining (mins) :  14.555528104305267

EPOCH               : 56  of  100
Validation accuracy : 0.993369038663

Epoch processing time (secs)    :  19.127541542053223
Total elapsed time (mins)       :  18.223705410957336
Estimated time remaining (mins) :  14.026863797505696

EPOCH               : 57  of  100
Validation accuracy : 0.9915837798

Epoch processing time (secs)    :  19.823358058929443
Total elapsed time (mins)       :  18.554118804136913
Estimated time remaining (mins) :  14.206739942232767

EPOCH               : 58  of  100
Validation accuracy : 0.991583779648

Epoch processing time (secs)    :  19.09190082550049
Total elapsed time (mins)       :  18.872340905666352
Estimated time remaining (mins) :  13.364330577850343

EPOCH               : 59  of  100
Validation accuracy : 0.99311400153

Epoch processing time (secs)    :  19.10797953605652
Total elapsed time (mins)       :  19.19083285331726
Estimated time remaining (mins) :  13.05711934963862

EPOCH               : 60  of  100
Validation accuracy : 0.989288446977

Epoch processing time (secs)    :  19.533735752105713
Total elapsed time (mins)       :  19.51642246643702
Estimated time remaining (mins) :  13.022490501403809

EPOCH               : 61  of  100
Validation accuracy : 0.990308594746

Epoch processing time (secs)    :  19.135277032852173
Total elapsed time (mins)       :  19.83536604245504
Estimated time remaining (mins) :  12.437930071353913

EPOCH               : 62  of  100
Validation accuracy : 0.992603927722

Epoch processing time (secs)    :  19.502218008041382
Total elapsed time (mins)       :  20.160421284039817
Estimated time remaining (mins) :  12.351404738426208

EPOCH               : 63  of  100
Validation accuracy : 0.993624075491

Epoch processing time (secs)    :  19.220635652542114
Total elapsed time (mins)       :  20.480789399147035
Estimated time remaining (mins) :  11.852725319067638

EPOCH               : 64  of  100
Validation accuracy : 0.992603927722

Epoch processing time (secs)    :  19.003814220428467
Total elapsed time (mins)       :  20.797547459602356
Estimated time remaining (mins) :  11.40228853225708

EPOCH               : 65  of  100
Validation accuracy : 0.992603927722

Epoch processing time (secs)    :  19.600350379943848
Total elapsed time (mins)       :  21.124245099226634
Estimated time remaining (mins) :  11.433537721633911

EPOCH               : 66  of  100
Validation accuracy : 0.992348890589

Epoch processing time (secs)    :  19.42087459564209
Total elapsed time (mins)       :  21.447950875759126
Estimated time remaining (mins) :  11.00516227086385

EPOCH               : 67  of  100
Validation accuracy : 0.994644223412

Epoch processing time (secs)    :  19.52510690689087
Total elapsed time (mins)       :  21.77339530388514
Estimated time remaining (mins) :  10.738808798789979

EPOCH               : 68  of  100
Validation accuracy : 0.994899260393

Epoch processing time (secs)    :  19.320788860321045
Total elapsed time (mins)       :  22.095432607332864
Estimated time remaining (mins) :  10.304420725504558

EPOCH               : 69  of  100
Validation accuracy : 0.993369038663

Epoch processing time (secs)    :  19.260403871536255
Total elapsed time (mins)       :  22.416463299592337
Estimated time remaining (mins) :  9.9512086669604

EPOCH               : 70  of  100
Validation accuracy : 0.995154297373

Epoch processing time (secs)    :  19.213947772979736
Total elapsed time (mins)       :  22.73672379652659
Estimated time remaining (mins) :  9.606973886489868

EPOCH               : 71  of  100
Validation accuracy : 0.992093853761

Epoch processing time (secs)    :  19.522294759750366
Total elapsed time (mins)       :  23.06212238073349
Estimated time remaining (mins) :  9.43577580054601

EPOCH               : 72  of  100
Validation accuracy : 0.994134149604

Epoch processing time (secs)    :  19.130465984344482
Total elapsed time (mins)       :  23.380991995334625
Estimated time remaining (mins) :  8.927550792694092

EPOCH               : 73  of  100
Validation accuracy : 0.995154297373

Epoch processing time (secs)    :  19.36742639541626
Total elapsed time (mins)       :  23.70380860964457
Estimated time remaining (mins) :  8.715341877937316

EPOCH               : 74  of  100
Validation accuracy : 0.993369038511

Epoch processing time (secs)    :  19.40251588821411
Total elapsed time (mins)       :  24.027208105723062
Estimated time remaining (mins) :  8.407756884892782

EPOCH               : 75  of  100
Validation accuracy : 0.99285896455

Epoch processing time (secs)    :  19.10182809829712
Total elapsed time (mins)       :  24.345596464474998
Estimated time remaining (mins) :  7.959095040957133

EPOCH               : 76  of  100
Validation accuracy : 0.991838816932

Epoch processing time (secs)    :  19.409529447555542
Total elapsed time (mins)       :  24.669114565849306
Estimated time remaining (mins) :  7.763811779022217

EPOCH               : 77  of  100
Validation accuracy : 0.993879112471

Epoch processing time (secs)    :  19.350805282592773
Total elapsed time (mins)       :  24.991652504603067
Estimated time remaining (mins) :  7.4178086916605634

EPOCH               : 78  of  100
Validation accuracy : 0.993879112471

Epoch processing time (secs)    :  19.303417682647705
Total elapsed time (mins)       :  25.313399529457094
Estimated time remaining (mins) :  7.077919816970825

EPOCH               : 79  of  100
Validation accuracy : 0.99030859505

Epoch processing time (secs)    :  19.283599615097046
Total elapsed time (mins)       :  25.634812406698863
Estimated time remaining (mins) :  6.749259865283966

EPOCH               : 80  of  100
Validation accuracy : 0.995409334353

Epoch processing time (secs)    :  19.13960099220276
Total elapsed time (mins)       :  25.953831632932026
Estimated time remaining (mins) :  6.37986699740092

EPOCH               : 81  of  100
Validation accuracy : 0.995664371486

Epoch processing time (secs)    :  19.5781307220459
Total elapsed time (mins)       :  26.280157498518626
Estimated time remaining (mins) :  6.199741395314534

EPOCH               : 82  of  100
Validation accuracy : 0.993624075643

Epoch processing time (secs)    :  19.575443506240845
Total elapsed time (mins)       :  26.606440762678783
Estimated time remaining (mins) :  5.872633051872254

EPOCH               : 83  of  100
Validation accuracy : 0.994644223564

Epoch processing time (secs)    :  19.952373504638672
Total elapsed time (mins)       :  26.939005851745605
Estimated time remaining (mins) :  5.653172492980957

EPOCH               : 84  of  100
Validation accuracy : 0.996174445295

Epoch processing time (secs)    :  19.594515323638916
Total elapsed time (mins)       :  27.265603240331014
Estimated time remaining (mins) :  5.225204086303711

EPOCH               : 85  of  100
Validation accuracy : 0.994134149452

Epoch processing time (secs)    :  19.08156967163086
Total elapsed time (mins)       :  27.58365387916565
Estimated time remaining (mins) :  4.770392417907715

EPOCH               : 86  of  100
Validation accuracy : 0.995409334505

Epoch processing time (secs)    :  19.38999080657959
Total elapsed time (mins)       :  27.906846785545348
Estimated time remaining (mins) :  4.524331188201904

EPOCH               : 87  of  100
Validation accuracy : 0.993369038511

Epoch processing time (secs)    :  19.469302654266357
Total elapsed time (mins)       :  28.231361564000448
Estimated time remaining (mins) :  4.218348908424377

EPOCH               : 88  of  100
Validation accuracy : 0.994644223412

Epoch processing time (secs)    :  19.479734897613525
Total elapsed time (mins)       :  28.556047546863557
Estimated time remaining (mins) :  3.895946979522705

EPOCH               : 89  of  100
Validation accuracy : 0.992348890589

Epoch processing time (secs)    :  19.331088304519653
Total elapsed time (mins)       :  28.878256769975028
Estimated time remaining (mins) :  3.544032855828603

EPOCH               : 90  of  100
Validation accuracy : 0.994389186432

Epoch processing time (secs)    :  19.069499969482422
Total elapsed time (mins)       :  29.196109565099082
Estimated time remaining (mins) :  3.178249994913737

EPOCH               : 91  of  100
Validation accuracy : 0.993879112471

Epoch processing time (secs)    :  19.162726879119873
Total elapsed time (mins)       :  29.5155104358991
Estimated time remaining (mins) :  2.874409031867981

EPOCH               : 92  of  100
Validation accuracy : 0.993369038663

Epoch processing time (secs)    :  19.61098599433899
Total elapsed time (mins)       :  29.842385200659432
Estimated time remaining (mins) :  2.614798132578532

EPOCH               : 93  of  100
Validation accuracy : 0.995409334505

Epoch processing time (secs)    :  19.140515327453613
Total elapsed time (mins)       :  30.16142019033432
Estimated time remaining (mins) :  2.233060121536255

EPOCH               : 94  of  100
Validation accuracy : 0.994389186432

Epoch processing time (secs)    :  19.390989065170288
Total elapsed time (mins)       :  30.48462828795115
Estimated time remaining (mins) :  1.9390989065170288

EPOCH               : 95  of  100
Validation accuracy : 0.995664371334

Epoch processing time (secs)    :  19.111520290374756
Total elapsed time (mins)       :  30.80318056344986
Estimated time remaining (mins) :  1.592626690864563

EPOCH               : 96  of  100
Validation accuracy : 0.995664371334

Epoch processing time (secs)    :  19.124191522598267
Total elapsed time (mins)       :  31.121941538651786
Estimated time remaining (mins) :  1.274946101506551

EPOCH               : 97  of  100
Validation accuracy : 0.995409334505

Epoch processing time (secs)    :  19.186110258102417
Total elapsed time (mins)       :  31.44173647960027
Estimated time remaining (mins) :  0.9593055129051209

EPOCH               : 98  of  100
Validation accuracy : 0.995664371334

Epoch processing time (secs)    :  19.126375436782837
Total elapsed time (mins)       :  31.76053417523702
Estimated time remaining (mins) :  0.6375458478927613

EPOCH               : 99  of  100
Validation accuracy : 0.99311400153

Epoch processing time (secs)    :  19.263559818267822
Total elapsed time (mins)       :  32.08161826531092
Estimated time remaining (mins) :  0.3210593303044637

EPOCH               : 100  of  100
Validation accuracy : 0.993879112471

Epoch processing time (secs)    :  20.178837776184082
Total elapsed time (mins)       :  32.41795717080434
Estimated time remaining (mins) :  0.0

Saved model as *signs_dropout_150*
Final Test, Set #2; basic data, equalised images, drop-out in pipeline, equalised *test* images
Using drop-out model
Test Accuracy: 0.9275534152984619
In [62]:
# Check this new model against the test image set
# This is a set of code which starts from a complete reset of the tensorflow graph
# Only the call to the function with sets up the network structure is re-used
# Had some issues loading the saved session file before taking this approach
# (For some reason, the code used to reload the first 4 sessions above was failing with a key error.)

tf.reset_default_graph()

keep_prob = tf.placeholder(tf.float32) # probability to keep units
# Set up placeholders
X = tf.placeholder(tf.float32, (None,32,32,3))      # dimensions set for 32x32x3 images
y = tf.placeholder(tf.int32, (None))
one_hot_y = tf.one_hot(y, 43)                       # sized for 43 output classes

# Set up the training pipeline and optimiser
logits = SignIdentificationNetwork2(X)
cross_entropy = tf.nn.softmax_cross_entropy_with_logits(logits, one_hot_y)
loss_operation = tf.reduce_mean(cross_entropy)
optimizer = tf.train.AdamOptimizer(learning_rate = rate)      # using the Adam optimiser - built in, better than SGD
training_operation = optimizer.minimize(loss_operation)

# Set up the evaluation function and related operations
# correct prediction if prediction matches the one-hot value
correct_prediction = tf.equal(tf.argmax(logits, 1), tf.argmax(one_hot_y, 1))
accuracy_operation = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))

saver = tf.train.Saver()
with tf.Session() as sess:
    saver.restore(sess, '/home/neil/Documents/CarND-Traffic-Sign-Classifier-Project/signs_dropout_150')
    test_accuracy = sess.run(accuracy_operation, feed_dict={X: test_eq, y: y_test, keep_prob: 1.0})
    #test_accuracy = evaluate(validation_images_2, validation_labels_2) #sess.run(accuracy_operation, feed_dict={X: test_eq, y: y_test, keep_prob: 1.0})
print('Final Test, Set #2; basic data, equalised images, drop-out in pipeline, equalised *test* images')
print('Using drop-out model')
print('Test Accuracy: {}'.format(test_accuracy))
Final Test, Set #2; basic data, equalised images, drop-out in pipeline, equalised *test* images
Using drop-out model
Test Accuracy: 0.9275534152984619

Reflections on the final model training

The validation accuracy does not improve much after the first 40 or so epochs. In fact, validation accuracy values are observed to be repeated - suggesting that the model is oscillating between trained parameters (weights/biases) which are then resulting in only a few validation samples changing state between correctly or incorrectly classified. Given that this model also incorporates drop-out to avoid over-fitting, this suggests that the structure of the model has reached a practical limit for the given training set. Clearly, additional epochs of training are not going to significantly improve the model performance.

Possible further refinements might be to vary the batch size, or possibly change the learning rate over time (i.e. reduce the learning step size over the series of training epochs).

The final test set accuracy is 92.8%. The validation set accuracy was c.99%, so the lower test set accuracy is a little disappointing and may still suggest some over-fitting by the model.

Question 5

What approach did you take in coming up with a solution to this problem? It may have been a process of trial and error, in which case, outline the steps you took to get to the final solution and why you chose those steps. Perhaps your solution involved an already well known implementation or architecture. In this case, discuss why you think this is suitable for the current problem.

Answer:

The approach taken was to adopt the LeNet architecture, with modifications to apply this architecture to the particular context (colour images, larger number of classes). The LeNet architecture is known to be effective in classification of images (originally, single handprint digits 0-9), this was considered to be a good starting point for a classifier which is to operate in a more general sense.

Once the LeNet architecture was in place and appropriate pre-processing steps implemented (equalisation of the image brightnesses), a trial-and-error approach was taken to determine suitable parameters (e.g. number of epochs).

Because the training set was unbalanced and had images of varying levels of contrast, a number of variations on the training set images were created and several models trained. This allowed the different pre-process steps to be tested and a most likely candidate chosen (in this case, the equalisation pre-process improved the measured validation accuracy, whilst the augmentation with additional images did not make a significant improvement).

As the observed test set accuracy was lower than the validation set accuracy, the architecture was updated to include random drop-out of observations during the convolutional layers. This was intended to address the apparent over-fitting of the model. This was only partially successful, as the test set accuracy (92.8%) remains lower than the validation accuracy levels seen during training (c.99%). (This might also be due to some subtle difference between the training and test set images, but this has not been investigated.)


Step 3: Test a Model on New Images

Take several pictures of traffic signs that you find on the web or around you (at least five), and run them through your classifier on your computer to produce example results. The classifier might not recognize some local signs but it could prove interesting nonetheless.

You may find signnames.csv useful as it contains mappings from the class id (integer) to the actual sign name.

Implementation

Use the code cell (or multiple code cells, if necessary) to implement the first step of your project. Once you have completed your implementation and are satisfied with the results, be sure to thoroughly answer the questions that follow.

In [102]:
### Load the images and plot them here.
### Feel free to use as many code cells as needed.

# read in some sample images
new_images = []
image_list = ['Udacity-50sign.bmp', 'ClaytonWest-30sign.bmp', 'GantrySpeed-50limit.bmp', 'GermanRightTurn.bmp','RandomGerman1-DeadEnd.bmp', 'RandomGerman1-priority.bmp']
for i in image_list:
    ni = ndimage.imread(i)
    new_images.append(ni)
    plt.figure()
    plt.imshow(ni)

Notes on sample images

Udacity Image

The image above is the Udacity title page image for this assignment. It contains a nice 50mph speed limit sign.

Clayton West Image

The image above is a UK street scene. Although the 30mph limit sign is actually for the side-road, this sign is detected by the speed limit detection system used by BMW when passing on the main road and therefore the speed limit is deemed to be applied to the main road (which is actually a 40mph zone). Whilst the recognition of the sign is correct, the incorrect context identification applied by the BMW system shows an obvious limitation of camera-based/full frame sign identification.

Gantry Image

The image above is an example of a UK motorway gantry digital speed sign setup. Whilst the speed signs are clear, the format is red-on-black rather than black-on-white and therefore differs from the typical training examples.

Right Turn Image

The image above is from an internet search for typical German traffic signs. The image is angled, but this is not expected to be an issue for the identification process.

Ramdom Image

The image above is an example German street scene, taken from Google Maps. There are 2 signs, one priority sign and one dead-end.

Question 6

Choose five candidate images of traffic signs and provide them in the report. Are there any particular qualities of the image(s) that might make classification difficult? It could be helpful to plot the images in the notebook.

**Answer

The candidate images are described above and are mostly expected to be recognised with a high level of confidence.

The motorway gantry speed signs are expected to be a problem for classification, in that these electronic signs are a different format to typical speed signs (speed signs typically being black lettering on a white background with a red circle, rather than red lettering on a black background). It will be interesting to see how the classifier handles this variation in colour and whether the similar structure is sufficient to allow recognition at a reasonable level of confidence.

The side-road 30mph sign is an example where the sign can be recognised, but where the positioning of the sign is not going to be helpful for an autonomous vehicle. In this particular instance, the BMW speed-limit detection system (which is assumed to be using a forward-facing camera) detects the 30mph sign consistently - but this causes the vehicle to provide erroneous information to the driver, as the actual road speed limit is 40mph. A similar effect in a self-driving car might cause the car to brake unexpectedly (from the perspective of any following vehicles).

Besides the examples above, potential issues for classification could be lighting conditions (darkness, low sun/reflections) and the angle of the sign to the camera (giving rise to apparent distortions in the presentation of the sign vs the largely square-on training data). Examples of these types are not present in this small sampe of new images (although the right-turn sign is slightly angled away from the camera, this is not expected to be sufficient to cause a problem for classification).

In [108]:
### Run the predictions here.
### Feel free to use as many code cells as needed.

# new_images contains the list of new images, as ndimage arrays
# need to equalise the images, as per pre-processing used before training the model
new_eq = []
for idx in range(len(new_images)):
    new_eq.append(equalise_image(new_images[idx]))
    
# get the predictions

saver = tf.train.Saver()
with tf.Session() as sess:
    saver.restore(sess, '/home/neil/Documents/CarND-Traffic-Sign-Classifier-Project/signs_dropout_150')
    feed_dict = {X: new_eq, keep_prob: 1.0}
    classification = sess.run(tf.argmax(logits, 1), feed_dict)
    
print (classification) 
/home/neil/anaconda3/envs/CarND-Traffic-Sign-Classifier-Project/lib/python3.5/site-packages/skimage/exposure/exposure.py:63: UserWarning: This might be a color image. The histogram will be computed on the flattened image. You can instead apply this function to each color channel.
  warn("This might be a color image. The histogram will be "
[ 5  1 35 20 35 11]

Classification results

The results on the classification of the sample images are:

  1. Class #5: speed limit 80km/h - expected speed limit 50 (incorrect)
  2. Class #1: speed limit 30km/h - expected speed limit 30 (correct)
  3. Class #35: ahead only - expected speed limit 50 (incorrect)
  4. Class #20: dangerous curve to the right - expected right turn (incorrect)
  5. Class #35: ahead only - expected dead end (incorrect)
  6. Class #11: right-of-way at next intersection - expected right-of-way (correct)
In [130]:
for i in range(len(new_images)):
    fig = plt.figure()
    fig.suptitle('Input image #' + str(i+1)) 
    plt.imshow(new_images[i])
    # find a match
    for n in range(len(training_images_2)):
        if training_labels_2[n] == classification[i]:
            fig = plt.figure()
            fig.suptitle('Example for class label: ' + str(classification[i])) 
            plt.imshow(training_images_2[n])
            break    

Question 7

Is your model able to perform equally well on captured pictures when compared to testing on the dataset? The simplest way to do this check the accuracy of the predictions. For example, if the model predicted 1 out of 5 signs correctly, it's 20% accurate.

NOTE: You could check the accuracy manually by using signnames.csv (same directory). This file has a mapping from the class id (0-42) to the corresponding sign name. So, you could take the class id the model outputs, lookup the name in signnames.csv and see if it matches the sign from the image.

Answer:

The model has correctly identified only 2 of 6 signs. Taking a look at each in turn, we see the following:

Image #1: 50 sign identified as an 80 sign. Although the 50 sign is reasonably clear, it is easy to see how the shape of the '5' can be recognised as an '8' and this would lead to the observed result.

Image #2: 30 sign identified correctly.

Image #3: digital 50 sign identified as ahead only sign. The digital sign is not in the training set (UK sign, not German), so it is no surprise that the sign has not been identified correctly. What is surprising is that the match does not appear at all similar, other than the circular shape.

Image #4: right turn recognised as a dangerous curve sign. These have different colours, but share the characteristic right-arrow shape.

Image #5: dead end recognised as ahead only. The dead end sign is not in the training set, although this is a genuine German roadsign (taken from a Google Streetview image from the Munich area). The shapes are similar.

Image #6: right of way sign identified correctly.

The model is only 33% accurate on these examples (2 of 6), or 50% accurate (2 of 4) if the two examples which are not in the training set are removed. This is slightly disappointing, given the high levels of accuracy seen in the training/validation/test processes (in excess of 90%).

In [123]:
### Visualize the softmax probabilities here.
### Feel free to use as many code cells as needed.
saver = tf.train.Saver()
with tf.Session() as sess:
    saver.restore(sess, '/home/neil/Documents/CarND-Traffic-Sign-Classifier-Project/signs_dropout_150')
    feed_dict = {X: new_eq, keep_prob: 1.0}
    softmax = sess.run(tf.nn.softmax(logits), feed_dict)
#print(len(softmax))
#print (softmax) 

# lets take a look at a barchart for the softmax values for the first sample image
N = len(softmax[0])
x = range(N)

fig = plt.figure()
ax = plt.subplot()
plt.bar(x, softmax[0])
ax.set_ylabel('Softmax probability')
ax.set_xlabel('Class Number')
ax.set_title('Softmax Probability')
Out[123]:
<matplotlib.text.Text at 0x7f11fb510be0>
In [128]:
# Experiment with the top_k function
saver = tf.train.Saver()
with tf.Session() as sess:
    saver.restore(sess, '/home/neil/Documents/CarND-Traffic-Sign-Classifier-Project/signs_dropout_150')
    feed_dict = {X: new_eq, keep_prob: 1.0}
    feed_dict = {X: new_eq, keep_prob: 1.0}
    sm = sess.run(tf.nn.softmax(logits), feed_dict)
    top_k = sess.run(tf.nn.top_k(sm, k=3))
print(top_k)
TopKV2(values=array([[  5.71790993e-01,   2.89016664e-01,   6.39818758e-02],
       [  8.98461521e-01,   6.91936985e-02,   2.18482055e-02],
       [  7.58644760e-01,   9.73976776e-02,   8.35525766e-02],
       [  8.57089221e-01,   8.12318549e-02,   4.00466695e-02],
       [  9.81772721e-01,   4.10065241e-03,   3.53658665e-03],
       [  9.97948706e-01,   1.84840464e-03,   8.27157201e-05]], dtype=float32), indices=array([[ 5,  7,  3],
       [ 1,  6,  5],
       [35, 38, 13],
       [20, 33, 40],
       [35, 25, 20],
       [11, 38, 42]], dtype=int32))

Question 8

Use the model's softmax probabilities to visualize the certainty of its predictions, tf.nn.top_k could prove helpful here. Which predictions is the model certain of? Uncertain? If the model was incorrect in its initial prediction, does the correct prediction appear in the top k? (k should be 5 at most)

tf.nn.top_k will return the values and indices (class ids) of the top k predictions. So if k=3, for each sign, it'll return the 3 largest probabilities (out of a possible 43) and the correspoding class ids.

Answer:

The bar-chart visualisation of the softmax probabilities for the first image show that the certainty was quite low - only just over 50%, with an alternative at c.30% probability.

The top_k values for each classification with k=3 shows the 3 most likely options for each image.

Image #1: This is the 50 speed sign mis-identified as an 80 sign. However, the second most likely option (class label 7) is for a 100 speed sign and the 3rd most likely is for a 60 sign. The actual 50 speed limit (class 2) was at a very low (barely non-zero probability).

Image #2: 30 sign identified correctly, with 89.8% certainty.

Image #3: digital 50 sign identified as ahead only sign. Second/third options were keep-right and yield. The ahead-only was given 75.9% confidence, which is very high, considering that this sign is not in the training data at all.

Image #4: right turn recognised as a dangerous curve sign, with 85.7% confidence. Second/third options were turn right and roundabout - hence the correct sign was the second option, but only with 8.6% confidence.

Image #5: dead end recognised as ahead only, with 98.2% confidence - very high. Second/third options were road work and dangerous curve, but at such low confidence levels as to not be valid candidates.

Image #6: right of way sign identified correctly with 99.8% confidence. The alternatives (keep right, end of no overtaking) are at very low confidence levels.

This analysis of confidence levels shows that the model is very confident with correct recognitions, but can also be very confident of incorrect identifications for signs which are not in the training set. It seems possible to use confidence levels to rule out marginal identification (i.e. it may be better for a self-driving car to not recognise a sign at all, or to know that it is not certain, rather than to act on incorrect information based on a relatively low confidence level for the classification).

What the introduction of new images seems to confirm is that the training set for a road-going implementation should contain a comprehensive representation of all signs actually used (the data set used for this project is incomplete, as shown by the dead-end sign extracted from a genuine German street scene). Ideally a future state would be establish a standard library and then legislation which would require new street signs to be included in this standard library as part of the introduction of that new street sign.

Note: Once you have completed all of the code implementations and successfully answered each question above, you may finalize your work by exporting the iPython Notebook as an HTML document. You can do this by using the menu above and navigating to \n", "File -> Download as -> HTML (.html). Include the finished document along with this notebook as your submission.

In [ ]: